Three-dimensional modeling method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a three-dimensional modeling method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of three-dimensional modeling. The three-dimensional modeling method comprises the following steps: based on multiple frames of original images corresponding to a target scene, obtaining an initial three-dimensional model corresponding to the target scene and camera parameters corresponding to the original images; inputting the initial three-dimensional model and the camera parameters into a neural rendering model to obtain target texture data output by the neural rendering model; and processing the low-surface-number three-dimensional model corresponding to the initial three-dimensional model based on the target texture data to obtain a target three-dimensional model corresponding to the target scene. According to the three-dimensional modeling method, secondary creation does not need to be carried out manually to improve the image quality, the labor cost is saved, the volume of data needing to be processed by the neural rendering model is reduced, and the high-fidelity rendering effect is achieved on the basis of ensuring the image fluency.

Description

Technical Field

[0001] This application belongs to the field of 3D modeling technology, and in particular relates to a 3D modeling method, apparatus, electronic device, storage medium and program product. Background Technology

[0002] Currently, digital twin technology, primarily offering immersive experiences, typically presents audiences with realistic 3D scenes through real-time rendering. The underlying technology supporting this real-time rendering relies on 3D modeling, and most of the model data used to construct immersive scenes comes from real-world 3D modeling data. Therefore, achieving high-fidelity representation of the geometry and textures of real-world models is a crucial data foundation for supporting immersive experiences. While methods using CG (computer graphics) exist, this approach requires processing massive amounts of data for immersive experiences, resulting in large model sizes. Excessive content can cause scene lag when the engine is processing the data. Furthermore, improving the image quality of real-world models requires manual re-creation, incurring significant human resource costs. Summary of the Invention

[0003] This invention aims to solve at least one of the technical problems existing in the prior art. To this end, this invention proposes a 3D modeling method, device, electronic device, storage medium, and program product that eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for neural rendering models, and achieves high-fidelity rendering effects while ensuring smooth image playback.

[0004] Firstly, this application provides a three-dimensional modeling method, including: Based on multiple frames of original images corresponding to the target scene, an initial 3D model corresponding to the target scene and camera parameters corresponding to each of the original images are obtained. The initial 3D model and the camera parameters are input into the neural rendering model to obtain the target texture data output by the neural rendering model; Based on the target texture data, the low-face-count 3D model corresponding to the initial 3D model is processed to obtain the target 3D model corresponding to the target scene.

[0005] According to the 3D modeling method provided in the embodiments of this application, a high-polygon initial 3D model is input into a neural rendering model to generate high-quality target texture data. Then, based on the target texture data, a low-polygon 3D model corresponding to the initial 3D model is processed by texture mapping to obtain the target 3D model. This eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for processing by the neural rendering model, and achieves high-fidelity rendering effect while ensuring smoothness of the image.

[0006] One embodiment of the 3D modeling method of this application, wherein obtaining the target texture data output by the neural rendering model includes: The initial 3D model is divided into multiple sub-surface patches using meta-representation, and a latent code corresponding to each sub-surface patch is generated. The latent code is processed based on the camera parameters to obtain the theoretical rendering color corresponding to the target texture data; The target texture data is obtained based on the theoretically rendered color and the camera parameters.

[0007] One embodiment of the three-dimensional modeling method of this application includes generating the latent code corresponding to each of the sub-surface patches, comprising: Generate the initial latent code corresponding to each of the sub-surface patches; Based on the camera parameters, the initial latent codes are corrected using metamorphic manifolds to obtain the latent codes.

[0008] One embodiment of the 3D modeling method of this application, wherein the method obtains each of the initial latent codes by correcting the camera parameters using metamorphic manifolds, includes: Based on the coordinate information corresponding to each of the sub-surface patches and the camera ray direction, the offset information corresponding to each of the initial latent codes is calculated; the camera ray direction is determined based on the camera parameters. Based on the offset information, the initial latent codes are offset to obtain the meta-deformation embedding; The latent code is obtained by performing surface mapping on the meta-deformation embedding.

[0009] One embodiment of the 3D modeling method of this application, wherein processing the latent code based on the camera parameters to obtain the theoretical rendering color corresponding to the target texture data includes: The latent code is decoded to obtain the geometric attribute information corresponding to the sub-surface patch; the geometric attribute information includes surface normal, diffuse color, specular albedo, and high-dimensional spatial features. The specular reflectance is calculated based on the surface normal, the specular albedo, and the high-dimensional space features. The theoretical rendering color is obtained based on the diffuse color, the specular reflectance, and the camera parameters.

[0010] One embodiment of the 3D modeling method of this application, wherein obtaining the target texture data based on the theoretical rendering color and the camera parameters includes: At least one intersecting triangle is obtained between the camera ray direction and the initial 3D model; the intersection point of each intersecting triangle corresponds to a theoretical rendering color; the camera ray direction is determined based on the camera parameters; At least one of the theoretical rendering colors is aggregated to obtain the target rendering color corresponding to the camera ray direction; The target texture data is obtained based on the target rendering color.

[0011] One embodiment of the 3D modeling method of this application includes processing a low-facet 3D model corresponding to the initial 3D model based on the target texture data to obtain a target 3D model corresponding to the target scene, comprising: Based on the target texture data, a target texture map is generated; The target texture map is added to the low-polygon 3D model to obtain the target 3D model.

[0012] Secondly, this application provides a three-dimensional modeling apparatus, comprising: The first processing module is used to obtain an initial 3D model corresponding to the target scene and camera parameters corresponding to each of the original images based on multiple frames of original images corresponding to the target scene. The second processing module is used to input the initial 3D model and the camera parameters into the neural rendering model to obtain the target texture data output by the neural rendering model. The third processing module is used to process the low-face-count 3D model corresponding to the initial 3D model based on the target texture data to obtain the target 3D model corresponding to the target scene.

[0013] According to the three-dimensional modeling apparatus provided in the embodiments of this application, a high-polygon initial three-dimensional model is input into a neural rendering model to generate high-quality target texture data. Then, based on the target texture data, a low-polygon three-dimensional model corresponding to the initial three-dimensional model is processed by texture mapping to obtain the target three-dimensional model. This eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for processing by the neural rendering model, and achieves high-fidelity rendering effect while ensuring smoothness of the image.

[0014] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional modeling method as described in the first aspect above.

[0015] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional modeling method as described in the first aspect above.

[0016] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional modeling method as described in the first aspect above.

[0017] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: By inputting a high-polygon initial 3D model into a neural rendering model to generate high-quality target texture data, and then performing texture mapping on a low-polygon 3D model corresponding to the initial 3D model based on the target texture data, the target 3D model is obtained. This eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for the neural rendering model to process, and achieves high-fidelity rendering effects while ensuring smooth visuals.

[0018] Furthermore, by designing a low-polygon, high-quality model production process, the data volume problem of high-quality real-world models was solved, reducing the rendering pressure on the engine for high-polygon, high-quality models. In addition, a "neural network rendering" process was incorporated into the original real-world modeling process, realizing the toolization of low-polygon, high-quality models.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the three-dimensional modeling method provided in the embodiments of this application; Figure 2 This is one of the schematic diagrams illustrating the principle of the three-dimensional modeling method provided in the embodiments of this application; Figure 3 This is the second schematic diagram illustrating the principle of the three-dimensional modeling method provided in the embodiments of this application; Figure 4 This is the third schematic diagram illustrating the principle of the three-dimensional modeling method provided in the embodiments of this application; Figure 5 This is the fourth schematic diagram illustrating the principle of the three-dimensional modeling method provided in the embodiments of this application; Figure 6 This is a second schematic flowchart of the three-dimensional modeling method provided in the embodiments of this application; Figure 7 This is the third flowchart illustrating the three-dimensional modeling method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the task management system provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the three-dimensional modeling device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] The following description, in conjunction with the accompanying drawings, details the three-dimensional modeling method, three-dimensional modeling device, electronic device, and readable storage medium provided in the embodiments of this application through specific examples and application scenarios.

[0024] Among them, the 3D modeling method can be applied to the terminal, specifically executed by the hardware or software in the terminal.

[0025] The three-dimensional modeling method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the three-dimensional modeling method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The three-dimensional modeling method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0026] like Figure 1 As shown, the 3D modeling method includes steps 110, 120 and 130.

[0027] Step 110: Based on the multiple original images corresponding to the target scene, obtain the initial 3D model corresponding to the target scene and the camera parameters corresponding to each original image; In this step, the target scene can be a historical building, ancient artifact, or geographical environment, etc.

[0028] It can collect data from multiple angles of the target scene to obtain multiple frames of original images, which can be pictures or videos.

[0029] It is possible to perform initial 3D reconstruction on multiple frames of original images to obtain a high-polygon 3D model, i.e., the initial 3D model.

[0030] Camera parameters can include position and rotation angle, i.e., camera attitude.

[0031] Step 120: Input the initial 3D model and camera parameters into the neural rendering model to obtain the target texture data output by the neural rendering model; In this step, the neural rendering model can generate view-dependent rendering effects, i.e., target texture data, based on each viewpoint defined by the camera parameters.

[0032] The target texture data is matched with the camera parameters corresponding to the original image.

[0033] By inputting an initial 3D model with a high polygon count into a neural rendering model, high-quality enhanced texture data, i.e., target texture data, can be obtained from the output of the neural rendering model.

[0034] Neural rendering models can be used in large-scale Novel View Synthesis (NVS) systems to recover underlying geometry using image resolution and visual... Figure 1 Using consistency as a boundary, implicit field synthesis with semantic representation is used to create a large baseline to ensure high fidelity of observations to details.

[0035] Step 130: Process the low-face-count 3D model corresponding to the initial 3D model based on the target texture data to obtain the target 3D model corresponding to the target scene.

[0036] In this step, the initial 3D model can be reduced in number of faces, reducing the high-face-count 3D model to a low-face-count 3D model.

[0037] The original image can be replaced with target texture data. For example, 3D software (such as Metashape) can be used to use the target texture data as a texture source to perform texture mapping calculations on a low-polygon 3D model and obtain the target 3D model.

[0038] Target texture data can be used to improve the visual quality of low-polygon 3D models.

[0039] Adding high-quality target texture data to a low-polygon 3D model with a small data volume results in a target 3D model with a small data volume and high image quality, which can be used in real-time rendering engines (such as game engines or digital twin platforms).

[0040] For example, the target 3D model can be applied to the digitization of cultural heritage and online museums, allowing users to virtually tour the site via mobile phone or computer browser.

[0041] During the research and development process, the inventors discovered that in the process of 3D modeling, in order to pursue high-quality rendering, it is necessary to acquire a large number of high-definition images, generate high-polygon meshes and high-resolution texture maps, resulting in huge model files, excessive load on the real-time rendering engine, scene lag, and poor user experience. In order to solve the above problems, manual modeling is often used to ensure smooth screen, thereby reducing the realism of the model. Improving the image quality of realistic models requires secondary creation by humans. Engineers need to produce a large number of handmade models and texture materials to achieve a high-quality realistic look, which is inefficient and has high labor costs.

[0042] In this application, a neural rendering model is introduced to decouple the geometric complexity and visual appearance of the model. It can generate a high-polygon initial 3D model based on multiple frames of original images of the target scene. Then, the neural rendering model is used to generate high-quality target texture data based on the high-polygon initial 3D model. The low-polygon 3D model corresponding to the initial 3D model is then textured based on the target texture data to obtain the target 3D model. This can significantly reduce the data volume and achieve high-fidelity rendering effect while ensuring the smoothness of the screen.

[0043] According to the 3D modeling method provided in the embodiments of this application, a high-polygon initial 3D model is input into a neural rendering model to generate high-quality target texture data. Then, based on the target texture data, a low-polygon 3D model corresponding to the initial 3D model is processed by texture mapping to obtain the target 3D model. This eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for processing by the neural rendering model, and achieves high-fidelity rendering effect while ensuring smoothness of the image.

[0044] In some embodiments, step 120 may include: Meta-representation is used to divide the initial 3D model into multiple sub-surface patches, and latent codes corresponding to each sub-surface patch are generated. The latent code is processed based on camera parameters to obtain the theoretical rendering color corresponding to the target texture data; Based on theoretical rendering colors and camera parameters, target texture data is obtained.

[0045] In this embodiment, "meta" refers to a locally defined surface manifold whose geometry and appearance are embedded in a surface manifold in a high-dimensional latent space.

[0046] Metas can be implicitly decoded into neural fields using MLP (i.e., implicit decoding). By deforming features, the geometric relationship of the viewpoint is mapped to the neural field, and the reflection field can be learned on the surface to generate a better viewpoint perception.

[0047] Meta is an NVS geometric representation that can represent geometry and appearance in a non-entangled way. Meta has the following properties: locality, which encodes scene geometry locally, reducing the difficulty of representing large-scale complex scenes; regularity, which accepts regular image structures and allows simple feature operations; and differentiability, which is suitable for mainstream gradient-based optimization techniques and provides efficient estimation of the underlying geometry.

[0048] like Figure 2 As shown, by using an implicit mapping function modeled by a multilayer perceptron, geometric and material properties are locally represented on a set of surface patches. The new view can be synthesized by querying another shading function that implicitly expresses illuminance and reflectivity. Optimization is achieved by matching the synthesized rendering with the available observations through differentiable rendering.

[0049] like Figure 2 As shown, a set of billion-pixel-level images can be obtained from sparse views as input, and a model for synthesizing images from arbitrary invisible views can be trained. Meta-representation allows for regular, explicit, and previewable geometric encoding in the network and allows for effective optimization through an automatic decoder framework. Furthermore, this application establishes an image generation model suitable for the new representation, incorporating intermediate geometry into the final rendering.

[0050] The neural rendering model can divide the initial 3D model into multiple sub-surface patches and establish a local coordinate system for each patch, assigning a latent code to each patch at the corner of the coordinate system.

[0051] In some embodiments, generating the latent code corresponding to each sub-surface patch includes: Generate the initial latent code corresponding to each sub-surface patch; Based on camera parameters, the initial latent codes are corrected using metamorphic manifolds to obtain each latent code.

[0052] In this embodiment, the viewing direction can be calculated based on camera parameters, and then the latent code can be geometrically corrected based on the viewing direction to obtain the corrected latent code.

[0053] Based on the corrected latent code corresponding to a point, the geometric properties of that point can be decoded. Then, combined with the lighting angle determined by the camera parameters, the highlight of that point can be calculated to obtain the theoretical rendering color of that point.

[0054] A single pixel may cover multiple tiny triangles. The model can aggregate their respective theoretical rendering colors to obtain the final target rendering color corresponding to that pixel. Repeating this operation on all pixels in the original image can yield the target texture data.

[0055] In actual implementation, such as Figure 3 As shown, the target scene M consists of a set of sub-surface patches, where each sub-surface patch S is parameterized as a UV map a, and x represents the 3D coordinates of a point on the sub-surface patch S. It is a UV map The corresponding local parameter coordinates are given by n, where n is the normal vector of each. Indicates the output view direction. express Reflection about n.

[0056] like Figure 3 As shown, the surface of the target scene can be reconstructed using a multi-view stereo method, and the mesh can be divided into a set of local surface patches S (i.e., sub-surface maps). Then, surface parameterization is performed on each sub-surface map S, that is, a planar continuous function is found. Assign (u, v) planar coordinates to each 3D point on S.

[0057] By parametrically mapping f and constructing a UV coordinate system (one coordinate system for each sub-surface map), various geometric properties of S can be encoded, such as surface normals, displacement, albedo, roughness, and metallic appearance.

[0058] The aforementioned local parameterized geometry can be transformed into a more compact neural representation, facilitating final optimization of the autodecoder framework.

[0059] For each sub-surface texture S, use Represent their respective UV coordinates, and further standardize them within a unit square. Defined as in A set of learnable high-dimensional eigenvectors at the four corners.

[0060] For a local parameter coordinate Using bilinear interpolation functions The corresponding intrinsic codes are obtained. The following equation applies:

[0061] in, For intrinsic codes, It is a bilinear interpolation function. For local parameter coordinates, These are high-dimensional eigenvectors.

[0062] It should be noted that, The rule structure supports the above simple operations, which are invalid for irregular representations such as point clouds and meshes.

[0063] Using coordinate-based multilayer perceptron neural networks (MLP) To decode The geometric properties are expressed by the following equation:

[0064] in, for Points in The selected geometric properties, It is a coordinate-based multilayer perceptron neural network (surface mapping network). For intrinsic codes, It is a bilinear interpolation function. For local parameter coordinates, These are high-dimensional eigenvectors.

[0065] point The geometric properties on use their parameterized coordinates and related intrinsic codes Decoding allows for optimization of surface properties through meta-representation. By rigorously learning radiation (or reflectivity) on a well-defined manifold, network capacity can be better allocated, network training can be stabilized, and implicit functions can express more detailed information.

[0066] A regularization term can be added to the training loss to achieve texture-based representation continuity, making the optimized normals on each texture close to the initial reconstructed surface normals. The regularization term can promote consistency between the optimized normals and the texture boundaries.

[0067] In some embodiments, based on camera parameters, each initial latent code is corrected using a metamorphic manifold to obtain each latent code, which may include: Based on the coordinate information corresponding to each sub-surface patch and the camera ray direction, the offset information corresponding to each initial potential code is calculated; the camera ray direction is determined based on camera parameters. Based on the offset information, each initial latent code is offset to obtain the meta-deformation embedding; The latent code is obtained by performing surface mapping on the meta-deformation embedding.

[0068] In this embodiment, the camera ray direction can be the ray formed when the center point of the camera is pointing towards the model when the camera is taking a picture of the model.

[0069] The neural rendering model can calculate the latent code of a sub-surface patch based on the coordinate information of the patch and the direction of the camera ray through the meta-deformation network. It requires an offset information, and the meta-deformation embedding can be obtained based on the offset information.

[0070] Surface mapping networks can read meta-deformation embeddings to obtain the corrected latent code.

[0071] In actual implementation, the parameterized points can be calculated based on the following formula. Along the camera ray direction color :

[0072] in, For parameterized points Along the camera ray direction The color, It is a diffuse color. The reflectivity is the specular reflectance. For lighting networks, It is a mirror view vector. For spatial features, The direction of the camera ray.

[0073] in:

[0074] in, It is the normal vector. It is a diffuse color. The reflectivity is the specular reflectance. For spatial features, For coordinate-based multilayer perceptron neural networks, For local parameter coordinates, For high-dimensional eigenvectors, It is a mirror view vector. To determine the output view direction, The direction of the camera ray.

[0075] Multiple views of the input image can be utilized Figure 1 The network is trained to obtain a viewpoint synthesis model that can acquire the initial geometry to restore high-frequency details in billion-pixel images.

[0076] like Figure 4 As shown, and For the local parameterized coordinates of the two shaded points, and For the corresponding potential embedding, For the direction of the camera ray, for of Predicted potential offset at the location, This is a meta-deformation embedding.

[0077] Solid curves represent the geometry of the underlying layer, while dashed lines represent inaccurate initial reconstructions.

[0078] During the research and development process, the inventors discovered that the rays in the diagram... and The rays reach the solid curve from different labels, due to inaccurate geometric estimation. and The same points were reached during the reconstruction process, therefore the same local parameterized points were used. Therefore, it is possible to optimize only the meta-representation. The labels that create contradictions.

[0079] On the other hand, light and They also reached the same point on the bottom curve. and The intersections associated with the reconstruction of dashed lines are different, which can lead to color differences in the reconstruction.

[0080] In this application, a metamorphic manifold is introduced, which can remap the initial geometry into an embedded latent space to implicitly promote the relevance of details.

[0081] For example, take and the corresponding parameterized points The goal is to find an offset in the latent space. ,so as well as The implicit coding is closer to the representation The intersection point between the curve and the underlying curve.

[0082] It can be used Relative to The meta-deformation embedding, in which, for The offset in the potential space.

[0083] Formally, the metamorphic manifold is parameterized as an MLP network. By implicitly modeling the microscopic geometric displacements in the latent space To compensate for imperfect geometry.

[0084] For each UV coordinate system, an additional potential geometric mesh is added for each sub-surface map S. .

[0085] Given a (u, v) coordinate and a camera ray direction We obtain the following equation:

[0086] in, for Hidden encoding of interpolation. For bilinear interpolation, For microscopic geometric displacement in potential space, For geometric latent vectors, For metamorphic networks, For local parameter coordinates, The direction of the camera ray.

[0087] In this application, the permutation geometry is embedded into a high-dimensional feature space for implicit deformation. The feature space can provide more useful parameter resetting and has more detailed feature measurement correlations.

[0088] The surface mapping network can then be increased using the following formula. Capacity:

[0089] That is, the modified surface mapping network requires material latent vectors. and geometric latent vector .

[0090] like Figure 4 As shown, the embedding of driving element deformation during the optimization process Achieved multi-view Figure 1 To, assumption If the reconstruction is of inaccurate local parameterized points, then from multiple perspectives (such as...) , )right Color observation can easily produce significant discrepancies.

[0091] Although when the direction of the ray is from arrive At that time, potential offset They can be different, but the meta-deformation embedding It always converges to the latent location (i.e., the entity curve), which explains the different observations with consistent cross-view colors (such as...). and ).

[0092] Modified surface mapping network Focus on The modeling represents the geometrically related properties, which is composed of... It shows the perspective of the lower class Figure 1 The surface exhibits unentangled properties after optimization.

[0093] In some embodiments, processing the latent code based on camera parameters to obtain the theoretical rendering color corresponding to the target texture data may include: The latent code is decoded to obtain the geometric attribute information corresponding to the sub-surface patch; The specular reflectance is calculated based on the surface normal, specular albedo, and high-dimensional space characteristics. The theoretical rendering color is obtained based on diffuse color, specular reflectance, and camera parameters.

[0094] In this embodiment, geometric attribute information of spatial variations can be predicted based on a surface mapping network. This geometric attribute information may include surface normals, diffuse color, specular reflectance, and high-dimensional spatial features, such as: ,in, For surface normal ( ), diffuse color ( ), Specular reflectance ( ), Features of high-dimensional space ( High-dimensional spatial features can be used to encode material information.

[0095] Specular reflectivity related to viewing angle can be estimated based on geometric property information.

[0096] In actual execution, the geometric attribute information and ray shading calculation formula can be derived based on the following steps. The interaction between geometric attributes in rendering can be determined based on the derivation process, and the neural network design can be guided based on the derivation process to simulate the entire rendering process.

[0097] Assuming the surface does not emit light and follows the rendering equation, let (x, n) be... Let a shaded point and its corresponding normal vector be given. For along The intensity of light in the x-direction is expressed as follows: Let r represent a unit vector point from x to the center of the camera. The opposite direction of the ray, i.e. .

[0098] Suppose a color equation related to the viewing angle The calculation method is as follows:

[0099] in, Let x be a unit vector point from the camera center. For surface normals.

[0100] It is the upper hemisphere ,and This is the bidirectional reflectance distribution function (BRDF). The BRDF can include a diffuse reflection term and a specular reflection term, as shown in the following equation:

[0101] Then decompose diffuse color Specular reflectivity related to viewing angle The following equation applies:

[0102] in:

[0103] Ignoring self-occlusion and mutual reflection, ignoring exposure changes and tone mapping, for lighting By performing pre-integration, we can obtain the approximate calculation formula as follows:

[0104] in, For specular reflectivity, The reflectivity is the specular reflectance. It is a mirror view vector. Let n be the direction of the viewpoint about the surface normal n. The reflection, that is , The pre-integrated illuminance is the specular component.

[0105] P predicted the specular albedo. To fit the viewpoint-dependent... A lighting network can be designed. This network consumes mirror view vectors. The spatial feature k is used to replace the shaded points, as shown in the following equation:

[0106] in,

[0107] In summary, the initially defined viewpoint-dependent color equation can be estimated using a neural network query, as shown in the following equation:

[0108] in, A color equation that is related to the viewing angle. It is a diffuse color. The reflectivity is the specular reflectance. It is a mirror view vector. It represents a high-dimensional spatial feature.

[0109] In this application, the rendering texture characteristics of neural networks are utilized to further refine the overall visual experience of the model, ensuring that the number of faces in the model does not affect the visual experience.

[0110] In some embodiments, the neural rendering model can be trained based on the following steps: The initial 3D model, multiple frames of original images, and camera parameters are used as a training set and input into the neural rendering model for iterative optimization. In each iteration, the color loss between the rendered color output by the neural rendering model and the pixel color of the original image is calculated. Based on color loss, the parameters of the neural rendering model are updated through the error backpropagation algorithm until the model converges, thus obtaining the neural rendering model.

[0111] In this embodiment, the neural rendering model may include a meta-deformation network, a surface mapping network, and a lighting network.

[0112] MLP can be used to predict latent spatial deformations, enhancing feature correspondences. Its inputs are geometric latent vectors, local parameter coordinates, and camera ray directions.

[0113] Scene attributes can be continuously expressed using surface mapping MLP, and lighting MLP can implicitly explain lighting and reflection inference.

[0114] like Figure 5 As shown, the local parameter coordinates can be... and camera ray direction Input to the meta-deformation network Combined with geometric latent vectors Metamorphic Network It can output the potential offset. Based on potential offset For geometric latent vectors By making corrections, we can obtain the corrected vector. .

[0115] Material latent vector with vector Decoding together allows us to obtain the geometric properties of the surface (surface normals). diffuse color Specular reflectance High-dimensional space features ).

[0116] Based on output view direction and surface normal The mirror view vector can be calculated using reflection operations. .

[0117] Combined with mirror view vector and features of high-dimensional space You can find the specular lighting. .

[0118] diffuse color With mirror lighting and specular reflectance By combining the calculation results, the final target rendering color can be obtained. .

[0119] It should be noted that sinusoidal activation and Film training modes can be used to transform geometric latent vectors. Material latent vectors and features of high-dimensional space It is passed into the neural network and used as an affine transformation oriented towards intermediate features to enhance the ability to represent high-frequency details.

[0120] The expression for the loss function can be:

[0121] in, A set of randomly selected camera viewpoints. For a batch of pixel rays randomly selected from viewpoint k, For the predicted rendered color, The colors are the original pixel colors of the real scene.

[0122] Constrained surface mapping networks can be used The normal prediction uses a coarse normal calculated from the input triangular mesh to guide the learning of geometric properties, as shown in the following equation:

[0123] in, Predict the normal of the spatial point where the pixel ray r of camera k intersects with the nearest triangle. The normal vector is derived from the input geometry.

[0124] The final loss consists of color loss and normal constraint:

[0125] in, For level smoothing weights, For color loss, For normal constraints.

[0126] In some embodiments, obtaining target texture data based on theoretically rendered colors and camera parameters may include: Obtain at least one intersecting triangle between the camera ray direction and the initial 3D model; Aggregate at least one theoretical rendering color to obtain the target rendering color corresponding to the camera ray direction; The target texture data is obtained based on the target rendering color.

[0127] In this embodiment, a camera ray direction can be calculated based on camera parameters, starting from the camera optical center and passing through pixels on the target texture data.

[0128] A patch on a grid may appear on multiple images, with one image corresponding to one camera and one ray direction corresponding to one theoretical rendering color.

[0129] It can obtain at least one intersecting triangle between the camera ray direction and the initial 3D model, and the intersection point of each intersecting triangle corresponds to a theoretical rendering color.

[0130] The target scene surface can be rasterized, and nearby predictions can be aggregated to generate a smooth scene rendering.

[0131] It is possible to trace pixel rays from the center of the camera, query neural networks at the intersection points to achieve view synthesis, and for each pixel ray of the camera, it can be determined as the target rendering color based on the average of a set of rasterized color predictions.

[0132] In practice, the task of view composition can be defined as learning a composite mapping. The expression is as follows:

[0133] in, For high-dimensional eigenvectors, For geometric latent vectors, For parameterized points Along the camera ray direction The color, For local parameter coordinates, The direction of the camera ray.

[0134] Metamorphic manifolds can continuously express local scene properties of a surface. Rendering and optimizing this neural representation requires a sampling mechanism: using the current differentiable rendering framework, the scene surface is rasterized with a z-buffer, and nearby predictions are merged to generate a smooth scene rendering.

[0135] In this way, a view synthesis is achieved by tracing pixel rays from the center of the camera and querying neural networks at the intersection points. For each pixel ray of camera k... Render colors It is calculated as the average of a set of rasterized color predictions, as shown in the following equation:

[0136] in, Let be the local parameter coordinates of the intersection point of the light rays on the i-th mesh triangle. This is the set of triangle indices selected from the z buffer.

[0137] During execution, you can select only the first one that is closest to the camera center. triangle, and from Delete the triangles that are far from the first intersection point in terms of depth value.

[0138] In some embodiments, step 130 may include: Generate a target texture map based on the target texture data; Add the target texture map to the low-polygon 3D model to obtain the target 3D model.

[0139] In this embodiment, the target texture data can be used as a new texture source and input into the 3D texture mapping software. The software can execute traditional texture mapping algorithms to obtain the target texture map.

[0140] Target texture mapping is used for visual compensation of low-polygon 3D models.

[0141] The target texture map contains high-frequency details, realistic material texture, and consistent lighting information learned and synthesized by the neural rendering model.

[0142] Adding a target texture map to a low-polygon 3D model can compensate for the lack of geometric details caused by the low polygon count.

[0143] In this application, by designing a model production process with low polygon count and high image quality, the problem of data volume of high-quality real-world models is solved, and the rendering pressure of the engine on high-real-world rendering image quality models is reduced; and a "neural network rendering" process is embedded into the original real-world modeling process, realizing the toolization of low polygon count and high image quality.

[0144] In actual implementation, such as Figure 6 As shown, the process flow of this application automates the process from data acquisition to high-quality rendering, and mainly includes the following three stages: Data acquisition and management phase: Initial data is collected through data acquisition scripts, and then stored and mapped to ensure data integrity and availability. Auxiliary measures include script optimization and data supplementation to further improve data quality.

[0145] Mesh construction phase: Initial data is used to generate the mesh and then validated. If necessary, point cloud validation is used to determine whether the data meets the requirements. If it does not meet the requirements, the parameters or topology are readjusted and the mesh is rebuilt.

[0146] Neural network rendering stage: The Mesh model is further refined and rendered. The model is finally rendered through a neural network, and high-quality visualization results are output. If adjustments are needed, the decision to perform low-poly topology or texture reconstruction is made based on the topology adjustment results.

[0147] like Figure 7 As shown, the task module design includes multiple task nodes with clear inputs and outputs. Through distributed nodes, process decoupling, and parameter passing, a complete task flow from 3D model creation to rendering can be realized.

[0148] Modeling and rendering tasks can be broken down into multiple nodes, and the task execution process can be completed through standardized parameter configuration and data transfer mechanisms.

[0149] Based on the actual needs of users, the task nodes can be divided into two main stages: the 3D modeling stage and the rendering stage; each node has clear inputs and outputs, and is decoupled and scheduled through task modules.

[0150] For example, the 3D modeling stage may include multiple task nodes: 3D_INIT (initialization), 3D_ALIGN (alignment), 3D_MODEL (model generation), 3D_TEXTURE (texture generation), and 3D_MODEL_OUT (model output).

[0151] 3D_INIT (Initialization): Input: Raw point cloud data or image file.

[0152] Output: Initialized data.

[0153] 3D_ALIGN (alignment): Input: Output data of 3D_INIT.

[0154] Output: Aligned point cloud file.

[0155] 3D_MODEL (Model Generation): Input: Alignment data output by 3D_ALIGN.

[0156] Output: The initial 3D Mesh model generated.

[0157] 3D_TEXTURE (texture generation): Input: Output of 3D_MODEL.

[0158] Output: The final model with textures added to the 3D Mesh.

[0159] 3D_MODEL_OUT (Model Output): Input: Output of 3D_TEXTURE.

[0160] Output: Model file (i.e., the initial 3D model, which can be an OBJ or FBX file), camera parameters, and deduplication images.

[0161] The rendering phase can include multiple task nodes: SURFEL_INIT (initialization), SURFEL_DATA_FORMAT (data formatting), SURFEL_BASE_DATA (basic data generation), SURFEL_WRAPPING (data optimization), SURFEL_SLICE (slicing), SURFEL_TRAIN (training), SURFEL_RENDER (rendering), SURFEL_METRICS (metric evaluation), and SURFEL_RENDER_OUT (rendering output).

[0162] SURFEL_INIT (Initialization): Input: 3D_MODEL_OUT Output data: Initial 3D model, camera parameters, and deduplication image.

[0163] Output: Metasurfel project file.

[0164] SURFEL_DATA_FORMAT (Data Formatting): Input: Output of SURFEL_INIT.

[0165] Output: Formatted model, camera, and image files.

[0166] SURFEL_BASE_DATA (Basic Data Generation): Input: Output of SURFEL_DATA_FORMAT.

[0167] Output: Basic data (such as normal map, depth map, etc.).

[0168] SURFEL_WRAPPING (Data Optimization): Input: Output of SURFEL_BASE_DATA.

[0169] Output: Optimized training data.

[0170] SURFEL_SLICE (slice): Input: Output of SURFEL_WRAPPING.

[0171] Output: Data slices (used for data validation).

[0172] SURFEL_TRAIN (training): Input: Output of SURFEL_SLICE.

[0173] Output: Statistical model training results.

[0174] SURFEL_RENDER (Rendering): Input: The trained model.

[0175] Output: Rendered image.

[0176] SURFEL_METRICS (Indicator Evaluation): Input: Output of SURFEL_RENDER.

[0177] Output: Rendering evaluation metrics.

[0178] SURFEL_RENDER_OUT (Render Output): Input: Rendering evaluation results (such as a rendered image verified by SURFEL_METRICS).

[0179] Output: A high-quality rendered image that can replace the original image of Metashape to generate a new texture (i.e., a target texture map).

[0180] In the task configuration interface, the running parameters of each node can be set during the initialization phase.

[0181] Running parameters can include alignment precision (3D_ALIGN), texture resolution (3D_TEXTURE), slice size (SURFEL_SLICE), and training steps (SURFEL_TRAIN).

[0182] Data can be transferred between nodes via files, such as point cloud files, model files, and texture images.

[0183] Scheduling parameters can be implemented through the task module, passed via API or configuration file.

[0184] Users can modify other parameters in the task configuration interface.

[0185] The task module can manage the status of all nodes in a unified manner. For example, the status (READY, RUNNING, FAILED, COMPLETED) can be set for each node.

[0186] When a node is in an abnormal state, the task module can trigger a retry and skip mechanism; users can manually adjust the node to the previous step.

[0187] Node scheduling and parameter coordination can be passed through the task module, and configuration can be done via file or API.

[0188] The task lifecycle supports pausing and resuming, retains data for completed parts of the task, and does not impose any restrictions on the task output.

[0189] In this application, by clearly defining input and output specifications, each node can operate independently and connect with other nodes through standardized data interfaces, achieving node decoupling in the task management module. It supports task pause, resumption, and retry functions, ensuring high reliability even in complex task flows and enabling dynamic management. Furthermore, it provides an intuitive task configuration interface and a real-time monitoring interface, helping users quickly grasp the task execution status and make necessary adjustments, thus improving the user experience. The modular design facilitates subsequent functional expansion, such as supporting more rendering algorithms or new modeling technologies, demonstrating high scalability. The integrated permission management module ensures strict control over task data access and operations, resulting in strong security.

[0190] All inputs, outputs, and intermediate data for a node can be stored on the local file system. Each task can have its root directory named after the task, and each node's data can be stored in a subdirectory named after the node. For example, for a task named "Task A", its data structure can be as follows: / tasks / TaskA / ├──3D_INIT / ├──3D_ALIGN / ├──3D_MODEL / ├──3D_TEXTURE / └──3D_MODEL_OUT / Meanwhile, the module design supports the synchronous cleanup of all storage files related to the task when the task is deleted, including logs and intermediate results.

[0191] like Figure 8 As shown, the task management system is located in the business layer and can work in conjunction with the presentation layer and the basic service layer.

[0192] The functional framework of a task management system can include: core functional modules and extended functional modules.

[0193] The core functional modules can be used for task management (creating, running, pausing, resuming and deleting tasks; supporting parallel execution of multiple tasks), digital asset management (associating tasks with digital assets and providing inputs for tasks (such as model files and point cloud data)), log management (recording the creation, running, completion or failure status of tasks; logs are persistently stored in the data layer), task result management (supporting the uploading (model files) and downloading (rendered graphs and generated models) of task results), and access control (implemented based on Sa-Token, with users divided into data acquisition personnel, task administrators and administrators, each with different operating permissions).

[0194] The extended functionality module can be used for task monitoring (real-time monitoring of task running status and providing front-end display) and result evaluation (providing three rendering quality evaluation metrics (mean psnr, mean ssim, mean lpips) and storing the evaluation results in the back-end).

[0195] Identity authentication and access control can be implemented based on Sa-Token. User roles can include data collection personnel, task administrators, and administrators, with operation permissions increasing progressively.

[0196] Parallel execution of multiple tasks can be achieved based on the following steps: Firstly, multiple tasks can be created. For example, the multi-threading mechanism of the Spring framework can be used to associate digital assets with tasks, and each task creation operation runs independently, ensuring thread safety.

[0197] Then, multiple tasks can be run. For example, each task can be started and executed through an external script or command-line tool (cmd) in a multi-process manner; the running status of the task is recorded in real time by the backend and dynamically refreshed on the frontend.

[0198] Then, parallel task scheduling is achieved based on the following steps: multiple tasks are executed simultaneously, and task queues and priority management are performed to ensure that high-priority tasks preempt resources.

[0199] In actual execution, for example, if the platform receives task A, task B and task C at the same time, the system backend can create an independent record for each task and store it in the task table, with the status of each task being "waiting".

[0200] When the task scheduling module detects that the system has sufficient computing resources, it can simultaneously set the status of three tasks to "in progress" and start three independent processing processes to perform 3D reconstruction and neural rendering in parallel. After the rendering work is completed, the status of each task can be updated to "completed".

[0201] In actual execution, the average processing time of each node should be controlled within a preset range (e.g., average alignment time of 3D_ALIGN < 5 minutes); and it should support more than 10 tasks running at the same time; the CPU and GPU utilization should be kept stable at around 70% to avoid resource waste or overload; and the latency from task creation to task start running should be controlled within 1 second.

[0202] When collaborating between the front-end and back-end, the front-end can submit task parameters (such as alignment precision and texture resolution) through the TaskCreate.vue page. After receiving the parameters, the back-end initializes the task, saves it to the database, and returns the task ID.

[0203] The backend can start the task's running process and record the task status through a log service. The frontend obtains the task's real-time status and logs through polling or WebSocket and dynamically updates the task details page.

[0204] The front-end can render task results (such as 3D models) using Three.js, provide a function to download task result files, and support the export of models and rendered images.

[0205] In this application, the task management system supports the parallel operation of multiple tasks, enabling batch model production, which can be applied to large-scale industrial scenarios and has strong commercialization capabilities.

[0206] The 3D modeling method provided in this application can be executed by a 3D modeling device. This application uses an example of a 3D modeling device executing the 3D modeling method to illustrate the 3D modeling device provided in this application.

[0207] This application also provides a three-dimensional modeling device.

[0208] like Figure 9 As shown, the 3D modeling device includes: a first processing module 910, a second processing module 920, and a third processing module 930.

[0209] The first processing module 910 is used to obtain an initial 3D model of the target scene and camera parameters corresponding to each original image based on multiple frames of original images corresponding to the target scene. The second processing module 920 is used to input the initial 3D model and camera parameters into the neural rendering model to obtain the target texture data output by the neural rendering model. The third processing module 930 is used to process the low-face-count 3D model corresponding to the initial 3D model based on the target texture data to obtain the target 3D model corresponding to the target scene.

[0210] According to the three-dimensional modeling apparatus provided in the embodiments of this application, a high-polygon initial three-dimensional model is input into a neural rendering model to generate high-quality target texture data. Then, based on the target texture data, a low-polygon three-dimensional model corresponding to the initial three-dimensional model is processed by texture mapping to obtain the target three-dimensional model. This eliminates the need for manual secondary creation to improve image quality, saves labor costs, reduces the data volume required for processing by the neural rendering model, and achieves high-fidelity rendering effect while ensuring smoothness of the image.

[0211] In some embodiments, the second processing module 920 may also be used for: Meta-representation is used to divide the initial 3D model into multiple sub-surface patches, and latent codes corresponding to each sub-surface patch are generated. The latent code is processed based on camera parameters to obtain the theoretical rendering color corresponding to the target texture data; Based on theoretical rendering colors and camera parameters, target texture data is obtained.

[0212] In some embodiments, the second processing module 920 may also be used for: Generate the initial latent code corresponding to each sub-surface patch; Based on camera parameters, the initial latent codes are corrected using metamorphic manifolds to obtain each latent code.

[0213] In some embodiments, the second processing module 920 may also be used for: Based on the coordinate information corresponding to each sub-surface patch and the camera ray direction, the offset information corresponding to each initial potential code is calculated; the camera ray direction is determined based on camera parameters. Based on the offset information, each initial latent code is offset to obtain the meta-deformation embedding; The latent code is obtained by performing surface mapping on the meta-deformation embedding.

[0214] In some embodiments, the second processing module 920 may also be used for: The latent code is decoded to obtain the geometric attribute information corresponding to the sub-surface patch; the geometric attribute information includes surface normal, diffuse color, specular reflectance and high-dimensional space features; The specular reflectance is calculated based on the surface normal, specular albedo, and high-dimensional space characteristics. The theoretical rendering color is obtained based on diffuse color, specular reflectance, and camera parameters.

[0215] In some embodiments, the second processing module 920 may also be used for: Obtain at least one intersecting triangle between the camera ray direction and the initial 3D model; the intersection point of each intersecting triangle corresponds to a theoretical rendering color; the camera ray direction is determined based on camera parameters; Aggregate at least one theoretical rendering color to obtain the target rendering color corresponding to the camera ray direction; The target texture data is obtained based on the target rendering color.

[0216] In some embodiments, the third processing module 930 can also be used for: Generate a target texture map based on the target texture data; Add the target texture map to the low-polygon 3D model to obtain the target 3D model.

[0217] The 3D modeling device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0218] The 3D modeling device in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0219] The three-dimensional modeling device provided in this application embodiment can achieve... Figures 1 to 8 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0220] In some embodiments, such as Figure 10As shown, this application embodiment also provides an electronic device 1000, including a processor 1001, a memory 1002, and a computer program stored on the memory 1002 and executable on the processor 1001. When the program is executed by the processor 1001, it implements the various processes of the above-described three-dimensional modeling method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0221] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0222] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described three-dimensional modeling method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0223] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0224] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described three-dimensional modeling method.

[0225] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0226] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described three-dimensional modeling method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0227] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0228] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0230] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0231] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0232] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A three-dimensional modeling method, characterized in that, include: Based on multiple frames of original images corresponding to the target scene, an initial 3D model corresponding to the target scene and camera parameters corresponding to each of the original images are obtained. The initial 3D model and the camera parameters are input into the neural rendering model to obtain the target texture data output by the neural rendering model; Based on the target texture data, the low-face-count 3D model corresponding to the initial 3D model is processed to obtain the target 3D model corresponding to the target scene.

2. The three-dimensional modeling method according to claim 1, characterized in that, The target texture data output by the neural rendering model includes: The initial 3D model is divided into multiple sub-surface patches using meta-representation, and a latent code corresponding to each sub-surface patch is generated. The latent code is processed based on the camera parameters to obtain the theoretical rendering color corresponding to the target texture data; The target texture data is obtained based on the theoretically rendered color and the camera parameters.

3. The three-dimensional modeling method according to claim 2, characterized in that, The generation of the latent code corresponding to each of the sub-surface patches includes: Generate the initial latent code corresponding to each of the sub-surface patches; Based on the camera parameters, the initial latent codes are corrected using metamorphic manifolds to obtain the latent codes.

4. The three-dimensional modeling method according to claim 3, characterized in that, The process of correcting each of the initial latent codes based on the camera parameters using metamorphic manifolds to obtain each of the latent codes includes: Based on the coordinate information corresponding to each of the sub-surface patches and the camera ray direction, the offset information corresponding to each of the initial latent codes is calculated; the camera ray direction is determined based on the camera parameters. Based on the offset information, the initial latent codes are offset to obtain the meta-deformation embedding; The latent code is obtained by performing surface mapping on the meta-deformation embedding.

5. The three-dimensional modeling method according to claim 2, characterized in that, The step of processing the latent code based on the camera parameters to obtain the theoretical rendering color corresponding to the target texture data includes: The latent code is decoded to obtain the geometric attribute information corresponding to the sub-surface patch; the geometric attribute information includes surface normal, diffuse color, specular albedo, and high-dimensional spatial features. The specular reflectance is calculated based on the surface normal, the specular albedo, and the high-dimensional space features. The theoretical rendering color is obtained based on the diffuse color, the specular reflectance, and the camera parameters.

6. The three-dimensional modeling method according to claim 2, characterized in that, The process of obtaining the target texture data based on the theoretically rendered color and the camera parameters includes: At least one intersecting triangle is obtained between the camera ray direction and the initial 3D model; the intersection point of each intersecting triangle corresponds to a theoretical rendering color; the camera ray direction is determined based on the camera parameters; At least one of the theoretical rendering colors is aggregated to obtain the target rendering color corresponding to the camera ray direction; The target texture data is obtained based on the target rendering color.

7. The three-dimensional modeling method according to any one of claims 1-6, characterized in that, The process of processing the low-facet 3D model corresponding to the initial 3D model based on the target texture data to obtain the target 3D model corresponding to the target scene includes: Based on the target texture data, a target texture map is generated; The target texture map is added to the low-polygon 3D model to obtain the target 3D model.

8. A three-dimensional modeling device, characterized in that, include: The first processing module is used to obtain an initial 3D model corresponding to the target scene and camera parameters corresponding to each of the original images based on multiple frames of original images corresponding to the target scene. The second processing module is used to input the initial 3D model and the camera parameters into the neural rendering model to obtain the target texture data output by the neural rendering model. The third processing module is used to process the low-face-count 3D model corresponding to the initial 3D model based on the target texture data to obtain the target 3D model corresponding to the target scene.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional modeling method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the three-dimensional modeling method as described in any one of claims 1-7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional modeling method as described in any one of claims 1-7.

Images