A method and system for extracting and synthesizing mesostructure texture based on Gaussian splash

Abstract

The application discloses a mesostructure texture extraction and synthesis method based on Gaussian splash, and the method comprises the following steps: constructing an embedded three-dimensional Gaussian splash of a grid and a Gaussian mixed texture representation in a tangent space of a two-dimensional basic grid, and decoupling a mesostructure texture from basic geometry; performing surface sampling on the grid manifold, extracting a neural texture block of the mesostructure, and applying a consistent fine-tuning strategy to eliminate discrete sampling artifacts; applying smooth regularization in a latent feature space, and synthesizing a neural texture from the extracted neural texture block. Through input of multi-view images containing mesostructure geometric details, the application can realize high-fidelity reconstruction, extraction and synthesis of mesostructure textures.

Description

Technical Field

[0001] This invention relates to computer graphics and computer vision, and more particularly to a neural implicit extraction, synthesis, and real-time rendering method for mesoscopic textures based on Gaussian splashing. Background Technology

[0002] In computer graphics, texture synthesis is crucial for enhancing the realism and diversity of 3D scenes. Traditional methods such as normal mapping, bump mapping, and displacement mapping effectively represent surface details using 2D textures. With advancements in deep learning, data-driven methods are becoming increasingly prominent in texture synthesis, offering significant advantages in terms of production diversity and visual fidelity.

[0003] To model the textures of mesostructures with fine-grained geometry, such as flowers and fences, from images acquired from the real world, existing techniques have introduced pipelines based on Neural Radiation Fields (NeRF). For example, NeRF-Texture proposes a method that combines Instant-NGP and Phong shading to reconstruct fine geometric details and synthesize high-resolution textures through block matching.

[0004] Problems with existing technologies include:

[0005] (1) Limitations of traditional two-dimensional methods

[0006] Traditional 2D texture mapping struggles to effectively represent volumetric mesoscopic textures with fine-grained geometry (such as flowers and fences). These structures exceed the representational limits of 2D texture mapping, and pipelines relying on manual modeling and texture processing become both laborious and inefficient for such complex texture details.

[0007] (2) Defects based on implicit neural radiation fields

[0008] To overcome the limitations of traditional methods, schemes based on implicit neural radiation fields (such as NeRF-Texture) offer a novel approach, but their practical application is still constrained by several factors. First, due to the spectral bias of low-frequency components in multilayer perceptrons (MLPs), these methods are insufficient for capturing high-frequency details. Second, their reliance on dense 3D sampling and heavy MLP inference makes real-time rendering impractical. Furthermore, the completely implicit nature of this representation hinders display control over synthesized textures, limiting its flexibility in practical applications.

[0009] In summary, there is an urgent need to design a novel mesh-Gaussian hybrid texture representation method that reconstructs a smooth base mesh through 2D Gaussian splashing and embeds neural texture features using tangent space to decouple mesoscopic textures from basic geometry. This method should also take into account high-frequency detail representation, controllability of texture synthesis, and real-time rendering efficiency, which is the core challenge currently facing this field. Summary of the Invention

[0010] The purpose of this invention is to address the technical bottlenecks in existing technologies when processing mesoscopic textures, which struggle to simultaneously capture high-frequency details and render in real time, and are unable to effectively decouple volumetric fine-grained textures from the underlying geometry. Furthermore, this invention aims to solve the distortion and artifact problems caused by frequency mismatch when sampling textures on discrete 3D Gaussian surfaces, and to address how to synthesize high-resolution textures that are structurally coherent, seamlessly stitched, and freely transferable to other 3D meshes.

[0011] In a first aspect, embodiments of this application provide a method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing, the method comprising the following steps:

[0012] Basic mesh extraction and preprocessing steps: Obtain a multi-view image dataset containing camera poses and a preprocessed contour mask. Reconstruct and optimize the basic geometry using two-dimensional Gaussian splashing to obtain a single-layer basic mesh.

[0013] Mesh and Gaussian blend texture representation construction steps: Construct an embedded 3D Gaussian splash of mesh and Gaussian blend texture representation in the tangent space of the 2D base mesh to build the mesh-Gaussian blend texture representation and decouple the mesoscopic structure texture from the basic geometry;

[0014] Consistency fine-tuning sampling steps: Surface sampling is performed on the mesh manifold to extract neural texture blocks of mesoscopic structures, and consistency fine-tuning is implemented through sampling consistency loss and reconstruction consistency loss to reduce discrete sampling artifacts;

[0015] Neural texture synthesis steps: Apply smoothing regularization to the latent feature space and synthesize neural textures from the extracted neural texture blocks.

[0016] Optionally, in this embodiment of the invention, the above-mentioned method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing further includes the following steps:

[0017] Rendering steps: The synthesized neural texture is transferred to the target 3D mesh through tangent space mapping, and multi-texture compositing and real-time rendering are performed.

[0018] Optionally, in this embodiment of the invention, the above-mentioned basic mesh extraction and preprocessing steps include:

[0019] The input multi-view original object mask is filtered to generate a coarse-grained contour mask;

[0020] Based on contour masking, the basic geometry is reconstructed using 2D Gaussian splashing, with mask loss used for supervision during the reconstruction process.

[0021] Based on various optimization coefficients, the mask loss, depth distortion loss, and normal consistency loss are jointly optimized.

[0022] The reconstructed geometry is subjected to a visibility masking process. Rays are projected from the center of the training camera onto the mesh surface to remove invisible internal occlusion surfaces and redundant geometric structures, ultimately resulting in a single-layer 2D base mesh with adaptive subdivision.

[0023] Optionally, in this embodiment of the invention, the above-mentioned mesh and Gaussian blend texture representation construction step includes:

[0024] Define neural implicit geometric features and appearance features on the base mesh; use barycentric coordinates to interpolate and calculate the surface normals and texture features of the coordinates;

[0025] The center coordinates of the 3D Gaussian are determined by predicting the offset of the Gaussian along the normal based on implicit geometric features using a shallow multilayer perceptron.

[0026] The Gaussian center is encoded, and multiple sets of multilayer perceptrons are used to regress the Gaussian splash parameters, including opacity, scaling, rotation, and color; and the Gaussian direction and view direction are defined in the local tangent space of the base mesh surface.

[0027] Optionally, in this embodiment of the invention, the above-mentioned consistency fine-tuning sampling step includes:

[0028] Within the user-specified area of ​​the basic mesh manifold, uniformly sample local square tangent sampling planes and record the local mesh frame and the sampling plane frame;

[0029] For each sampling plane, rays are projected from the pixels of the sampling plane along the surface normal direction and intersect the grid. Only when all rays intersect the grid within the specified area, the neural implicit texture features at the intersection point are queried by interpolation to form a locally structured neural texture block.

[0030] The sampling consistency loss is used to constrain the rendering results of the sampled 3D Gaussian generated by the neural texture block and the reconstructed 3D Gaussian after consistency fine-tuning to be consistent in the sampling view and the training view.

[0031] A reconstruction consistency loss is introduced to extract the original reconstruction quality before and after.

[0032] Optionally, in this embodiment of the invention, the above-mentioned neural texture synthesis step includes:

[0033] A smoothing regularization loss is constructed based on the difference in latent texture features between adjacent vertices in the base mesh to constrain the spatial continuity of latent texture features;

[0034] When performing block matching and stitching, a block matching algorithm with perceptual tiling is adopted, and a differentiable tiling metric is introduced as a weight. For the stitching region and candidate texture blocks, a conditional probability distribution is constructed based on similarity, and seamless neural texture synthesis is completed by iterative sampling according to the conditional probability distribution.

[0035] Optionally, in this embodiment of the invention, the above rendering step includes:

[0036] For a target mesh with surface parameterization, surface points are randomly sampled, and the tangent space information of the synthetic texture is obtained through nearest neighbor interpolation in order to calculate the mapping transformation from the local tangent space of the source object to the local tangent space of the target 3D mesh through the tangent sampling plane.

[0037] Based on the height factor, reference offset, and scaling factor, dynamically adjust the Gaussian center and scale on the target mesh;

[0038] Real-time rendering is achieved by performing differential rasterization on the updated Gaussian texture mapped to the target surface, and the height factor, reference offset, and scaling factor parameters are dynamically adjusted to achieve multi-texture composite generation.

[0039] Secondly, embodiments of this application provide a system for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing. Employing the aforementioned method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing, the system includes the following modules:

[0040] Basic Mesh Extraction and Preprocessing Module: Acquires a multi-view image dataset containing camera poses and a preprocessed contour mask, reconstructs and optimizes the basic geometry using 2D Gaussian splashing, and obtains a 2D basic mesh;

[0041] Mesh and Gaussian blend texture representation building module: Constructs an embedded 3D Gaussian splash in the tangent space of a 2D base mesh to build a mesh-Gaussian blend texture representation, decoupling the mesoscopic texture from the underlying geometry;

[0042] Consistency Fine-tuning Sampling Module: Performs surface sampling on the mesh manifold, extracts neural texture blocks of mesoscopic structures, and implements consistency fine-tuning through sampling consistency loss and reconstruction consistency loss to reduce discrete sampling artifacts;

[0043] Neural texture synthesis module: applies smoothing regularization to the latent feature space and synthesizes neural textures from the extracted neural texture blocks;

[0044] Rendering module: Through tangent space mapping, the synthesized neural textures are transferred to the target 3D mesh, and multi-texture compositing and real-time rendering are performed.

[0045] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing.

[0046] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing as described above.

[0047] Compared with existing technologies, it has the following outstanding advantages:

[0048] 1) The method and system of this invention propose a mesh-Gaussian hybrid texture representation method.

[0049] Technical Results: Abandoning the noisy moving cube algorithm, this method utilizes 2D Gaussian splashing to reconstruct a smooth and accurate coarse base mesh, defining neural implicit texture features in the local tangent space of this mesh. Explicit 3D Gaussian meta-parameters are regressed from these features, successfully achieving physical and logical decoupling between mesoscale texture details and the geometry of the underlying object. This representation overcomes the limitation of pure implicit representations being unable to render in real-time, supporting not only high-speed real-time rendering but also laying the foundation for subsequent texture mapping and independent transfer.

[0050] 2) The method and system of this invention propose a consistent fine-tuning sampling strategy.

[0051] Technical effect: When extracting local texture blocks directly from the mesh manifold surface, a joint loss constraint of sampling consistency and reconstruction consistency is introduced to force the alignment of the sampled latent Gaussian with the original reconstructed 3D Gaussian space. This mechanism effectively compensates for the interpolation blurring, distortion, or holes caused by the mismatch between the sampling frequency and the actual density of the discrete Gaussian, ensuring scale consistency and high-quality extraction of texture rendering at different sampling scales.

[0052] 3) The method and system of this invention propose a block synthesis algorithm that combines smoothing regularization and perceptual tiling.

[0053] Technical Effects: To ensure visual and structural coherence of the stitched areas, a smoothing regularization constraint is introduced into the latent feature space to promote spatial continuity of features and prevent sampling degradation. Combined with a block matching algorithm that perceives tiling, this invention can generate seamless, high-fidelity, high-resolution neural textures from extracted neural texture blocks. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a schematic diagram of the mesoscopic structure texture extraction and synthesis method based on Gaussian splashing according to the present invention;

[0056] Figure 2 This is a flowchart illustrating the overall technical solution of an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the basic mesh post-processing flow according to an embodiment of the present invention;

[0058] Figure 4 This is a comparison chart of the rendering quality of the present invention and existing technologies on a new perspective compositing task, as shown in the embodiments of the present invention.

[0059] Figure 5 A comparison of neural texture block extraction effects before and after introducing a consistency fine-tuning strategy in an embodiment of the present invention;

[0060] Figure 6 These are comparison images of the mesoscopic structure texture synthesis effects from different perspectives in embodiments of the present invention;

[0061] Figure 7a This is a schematic diagram illustrating the effect of cross-target mesh mapping and composite rendering of mesoscopic structure textures in an embodiment of the present invention. Figure 1 ;

[0062] Figure 7b This is a schematic diagram illustrating the effect of cross-target mesh mapping and composite rendering of mesoscopic structure textures in an embodiment of the present invention. Figure 2 ;

[0063] Figure 8 This is a schematic diagram of the mesoscopic structure texture extraction and synthesis system based on Gaussian splashing according to the present invention;

[0064] Figure 9 This is a schematic diagram of the computer hardware of the present invention. Detailed Implementation

[0065] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0066] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0067] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0068] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0071] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0073] While researching mesoscopic texture synthesis, the inventors discovered that existing technologies often struggle to achieve real-time rendering while maintaining high-fidelity texture details when handling fine-grained textures with complex geometries. Through in-depth analysis of implicit field-based methods, the inventors pointed out that their performance bottleneck lies in their over-reliance on dense 3D spatial sampling and massive MLP inference, which severely restricts the effective decoupling of texture from the underlying geometry. To address this, the inventors proposed a mesh-Gaussian hybrid texture representation method through extensive mesh processing experiments. This method reconstructs a smooth base mesh using 2D Gaussian splashing and successfully achieves logical separation between mesoscopic texture and underlying geometry by embedding neural texture features in tangent space.

[0074] Furthermore, the inventors discovered in their experiments that existing methods, when sampling surface textures based on discrete 3D Gaussian representations, are prone to severe interpolation blurring, holes, and spatial noise in the rendering results due to a mismatch between the sampling frequency and the actual Gaussian density. To overcome this challenge, the inventors proposed a "consistency fine-tuning" strategy through comparative experiments. This method effectively aligns the latent texture space and the reconstructed texture space by forcibly applying complementary constraints under both the sampling and training perspectives, thereby ensuring the consistency of rendering at different scales.

[0075] Finally, addressing the structural breaks and inconsistencies that arise during cross-object mapping and stitching of synthesized textures, the inventors, through research on the structural features of the feature space, proposed introducing smoothing regularization constraints into the latent feature space and combining this with a block matching algorithm that perceives tiling. This ultimately achieved high-fidelity, structurally coherent, and seamlessly stitched neural texture generation. This method not only significantly reduces the reliance on expensive implicit field queries and achieves real-time rendering performance, but also enables the synthesized neural textures to be flexibly mapped and rendered in real-time across topological meshes, achieving significantly better results than existing techniques on multiple real-world datasets.

[0076] The system of this application embodiment will be described in detail below with reference to specific embodiments:

[0077] like Figure 1 and Figure 2 As shown, this invention provides a neural implicit mesoscopic structure texture extraction and synthesis framework based on Gaussian splashing, which mainly includes three key methods: a mesh-Gaussian mixture texture representation method, a consistency fine-tuning extraction strategy, and a texture synthesis module that perceives tilingability.

[0078] like Figure 1 As shown, this invention provides a method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing. The method includes the following steps:

[0079] Basic mesh extraction and preprocessing step 101: Obtain a multi-view image dataset containing camera poses and a preprocessed contour mask. Reconstruct and optimize the basic geometry using two-dimensional Gaussian splashing to obtain a single-layer basic mesh.

[0080] Step 102 of constructing mesh and Gaussian blend texture representation: Construct an embedded 3D Gaussian splash of mesh and Gaussian blend texture representation in the tangent space of the 2D base mesh to construct the mesh-Gaussian blend texture representation and decouple the mesoscopic structure texture from the basic geometry;

[0081] Consistency fine-tuning sampling step 103: Perform surface sampling on the mesh manifold to extract neural texture blocks of mesoscopic structures, and implement consistency fine-tuning through sampling consistency loss and reconstruction consistency loss to reduce discrete sampling artifacts;

[0082] Neural texture synthesis step 104: Apply smoothing regularization to the latent feature space and synthesize neural textures from the extracted neural texture blocks.

[0083] Optionally, in this embodiment of the invention, the above-mentioned method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing further includes the following steps:

[0084] Rendering step 105: Through tangent space mapping, the synthesized neural texture is transferred to the target 3D mesh, and multi-texture compositing and real-time rendering are performed.

[0085] Optionally, in this embodiment of the invention, the basic mesh extraction and preprocessing step 101 includes:

[0086] The input multi-view original object mask is filtered to generate a coarse-grained contour mask. Based on the contour mask, the basic geometry is reconstructed using 2D Gaussian splashing, and the mask loss is used for supervision during the reconstruction process. Based on various optimization coefficients, the mask loss, depth distortion loss and normal consistency loss are jointly optimized. Visibility masking is performed on the reconstructed geometry. Rays are projected from the center of the training camera to the mesh surface to remove invisible internal occlusion surfaces and redundant geometric structures, finally obtaining an adaptively subdivided single-layer 2D basic mesh.

[0087] Optionally, in this embodiment of the invention, the above-mentioned mesh and Gaussian blend texture representation construction step 102 includes:

[0088] Neural implicit geometric features and appearance features are defined on the base mesh; the surface normal and texture features of the coordinates are interpolated using barycentric coordinates; the offset of the Gaussian along the normal is predicted based on the implicit geometric features using a shallow multilayer perceptron to determine the center coordinates of the 3D Gaussian; the center of the Gaussian is encoded, and the Gaussian splash parameters, including opacity, scaling, rotation, and color, are regressed using multiple sets of multilayer perceptrons; and the Gaussian direction and view direction are defined in the local tangent space of the base mesh surface.

[0089] Optionally, in this embodiment of the invention, the above-mentioned consistency fine-tuning sampling step 103 includes:

[0090] Within the user-specified area of ​​the basic mesh manifold, local square tangent sampling planes are uniformly sampled, and the local mesh frame and sampling plane frame are recorded. For each sampling plane, rays are projected from the pixels of the sampling plane along the surface normal direction to intersect the mesh. Only when all rays intersect the mesh within the specified area are the neural implicit texture features at the intersection points queried by interpolation to form locally structured neural texture blocks. A sampling consistency loss is used to constrain the rendering results of the sampled 3D Gaussian generated by the neural texture blocks and the reconstructed 3D Gaussian after consistency fine-tuning to be consistent in the sampling view and the training view. A reconstruction consistency loss is also introduced to extract the original reconstruction quality before and after.

[0091] Optionally, in this embodiment of the invention, the neural texture synthesis step 104 includes:

[0092] Smoothing regularization constraints are introduced for the latent features of the grid vertices. When performing block matching and stitching, a block matching algorithm with perceptual tiling is adopted, and a differentiable tiling metric is introduced as a weight. For the stitching region and candidate texture blocks, a conditional probability distribution is constructed based on similarity, and seamless neural texture synthesis is completed by iterative sampling according to the conditional probability distribution.

[0093] Optionally, in this embodiment of the invention, the rendering step 105 includes:

[0094] For a target mesh with surface parameterization, surface points are randomly sampled, and the tangent space information of the synthesized texture is obtained through nearest neighbor interpolation. This is used to calculate the mapping transformation from the local tangent space of the source object to the local tangent space of the target 3D mesh through the tangent sampling plane. Based on the height factor, reference offset, and scaling factor, the Gaussian center and scale on the target mesh are dynamically adjusted. The updated Gaussian mapped to the target surface is subjected to differential rasterization for real-time rendering, and the height factor, reference offset, and scaling factor parameters are dynamically adjusted to achieve multi-texture composite generation.

[0095] The specific embodiments of this application will now be described with reference to the accompanying drawings.

[0096] like Figure 2 As shown, the method for extracting and synthesizing mesoscopic structure textures based on 3D Gaussian splashing provided by this invention mainly includes the following steps:

[0097] S1: Input multi-view dataset and preprocessed contour mask, extract and optimize coarse base mesh;

[0098] S2: Construct a mesh-Gaussian hybrid texture representation in the tangent space of the base mesh to decouple the mesoscopic texture from the geometry;

[0099] S3: Extract neural texture blocks by surface sampling on the mesh manifold and apply a consistency fine-tuning strategy to eliminate discrete sampling artifacts;

[0100] S4: Apply smoothing regularization to the feature space and synthesize global high-resolution neural textures based on a block matching algorithm with perceptual tilingability;

[0101] S5: The synthesized neural textures are transferred to the target 3D mesh through tangent space mapping, and multi-texture compositing and real-time rendering are performed.

[0102] 1. For example Figure 3 As shown, the S1 basic mesh extraction and preprocessing method includes:

[0103] S11: Apply morphological filtering (including erosion, dilation, and blurring) to the input multi-view original object mask to generate a coarse contour mask. ;

[0104] S12: Based on The basic geometry is reconstructed using 2D Gaussian splashing. Mask loss is used during the reconstruction process. Supervision:

[0105]

[0106] in An alpha image rendered using 2D Gaussian rasterization. This is a coarse-grained contour mask obtained through image morphological filtering.

[0107] S13: Jointly optimize mask loss and depth distortion loss and normal consistency loss The overall objective function is:

[0108]

[0109] Among them, coefficient , , These are the weights for mask loss, depth distortion loss, and normal uniformity loss, respectively; in one embodiment, , , Set them to 0.5, 100, and 0.05 respectively;

[0110] S14: Perform visibility masking on the reconstructed geometry, i.e., project a ray from the center of the training camera onto the mesh surface to remove invisible internal occlusion surfaces. Then, convex decomposition and remeshing techniques are used to remove redundant geometry, ultimately obtaining an adaptively subdivided single-layer base mesh.

[0111] 2. For example Figure 2 As shown, the mesh-Gaussian blend texture representation in S2 is constructed as follows:

[0112] S21: Define neural implicit geometry features on the base mesh and appearance features Let the projection of the primitive element onto the k-th mesh surface be... Using barycentric coordinates its surface normal Interpolation calculations are performed on texture features.

[0113] S22: Predicting the Gaussian offset along the normal direction using a shallow multilayer perceptron based on implicit geometric features.

[0114] Thus, the center coordinates of the 3D Gaussian are determined:

[0115] ,in, Here are the parameters of the geometric multilayer perceptron used to predict the 3D Gaussian normal offset, and n(k,u,v) is the surface normal obtained by centroid interpolation at the projection point x(k,u,v).

[0116] S23: Encoding using HashGrid The Gaussian center is encoded, and the opacity of the Gaussian is regressed using multiple sets of multilayer perceptrons. Scaling Rotation And color c.

[0117] S24: Local tangent space on the surface of the base mesh The Gaussian direction and view direction are defined internally. During rendering, the Gaussian rotation matrix is ​​converted to world coordinates. The line of sight is converted into local coordinates. , This represents the direction of the line of sight in the world coordinate system.

[0118] 3. For example Figure 5 As shown, the surface sampling extraction and consistency fine-tuning of neural texture blocks in S3 includes:

[0119] S31: Within a user-specified region of the base mesh manifold, uniformly sample local square tangent sampling planes. Record the local mesh frame. With sampling plane frame This is to ensure alignment during subsequent texture mapping.

[0120] S32: For each sampling plane, project rays from its pixels along the surface normal direction to intersect the mesh. Only when all rays intersect the mesh within the specified region are locally structured neural texture blocks formed by interpolating the neural implicit texture features at the intersection points.

[0121] S33: To solve the problem of extracting sampling frequency With actual Gaussian density Distortion caused by mismatch introduces sampling consistency loss, forcing the sampled texture blocks to be Gaussian. With the original reconstruction of Gauss In sampling view and training view The rendering results remain consistent:

[0122] ,in, The image reconstruction loss consists of L1 loss and D-SSIM loss. This is the rendering function for view v.

[0123] S34: Introduce reconstruction consistency loss to maintain the original reconstruction quality before and after extraction.

[0124] ,in, For training view The corresponding real image, To reconstruct a 3D Gaussian before consistency fine-tuning, Reconstructed 3D Gaussian after consistency fine-tuning

[0125] 4. For example Figure 6 As shown, the neural texture synthesis for perceiving tilingability in S4 includes:

[0126] S41: To ensure structural continuity and prevent degradation during potential texture space matching, for mesh vertices... Potential characteristics Introduce smoothing regularization constraints:

[0127]

[0128] S42: When performing block matching and stitching, a block matching algorithm with perceptual tiling is adopted, and a differentiable tiling metric is introduced. As a weight. For the sutured area. With candidate texture blocks Based on similarity Construct conditional probability distributions, For each candidate texture block, the maximum similarity among the corresponding similarities is used to iteratively sample and complete the seamless neural texture synthesis according to this distribution:

[0129]

[0130] 5. For example Figure 7a and Figure 7b As shown, texture mapping and Gaussian splash rendering in S5 include:

[0131] S51: For a target mesh with surface parameterization, randomly sample surface points. The tangent space information of the synthesized texture is obtained through nearest-neighbor interpolation, and the mapping transformation from the source object to the target surface is calculated. .

[0132] S52: To compensate for scaling differences caused by mapping, a height factor is introduced. Reference offset and scaling factor , For predicting the three-dimensional Gaussian normal offset, a geometric multilayer perceptron is used. For regressing a three-dimensional Gaussian scale, a multilayer perceptron... , These are the neural implicit geometric features and appearance features mapped to points on the target surface, respectively. The surface normal at the target surface point. Multi-resolution hash grid encoding for the 3D Gaussian center coordinate p, dynamically adjusting the Gaussian center on the target grid. and scale :

[0133]

[0134]

[0135] S53: Perform differential rasterization on the updated Gaussian mapped to the target surface for real-time rendering, and dynamically adjust the above parameters through the GUI to achieve multi-texture composite generation.

[0136] As described above, the method of the present invention can be implemented well.

[0137] Compared with the prior art, the present invention has the following outstanding advantages and beneficial effects:

[0138] Compared with the prior art, the present invention has the following beneficial effects:

[0139] This invention enables high-fidelity reconstruction, extraction, and synthesis of mesoscopic textures by inputting multi-view images containing mesoscopic geometric details. Due to the high efficiency of explicit 3D Gaussian units, this method overcomes the computational burden of existing neural radiation field texture methods, achieving real-time rendering performance. Furthermore, the neural textures synthesized by this invention can be mapped onto any target 3D mesh with surface parameterization via tangent space transformation as needed, even supporting real-time composite rendering of multiple synthesized textures. This method demonstrates excellent texture fidelity and generalization ability on real-world datasets, providing a universal solution for the realistic texture generation and flexible editing of high-quality 3D digital assets.

[0140] Secondly, such as Figure 8 As shown, this application provides a system for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing. The system employs the aforementioned method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing, and includes the following modules:

[0141] Basic Mesh Extraction and Preprocessing Module 201: Obtains a multi-view image dataset containing camera poses and a preprocessed contour mask, reconstructs and optimizes the basic geometry using two-dimensional Gaussian splashing, and obtains a single-layer basic mesh;

[0142] Mesh and Gaussian mixture texture representation construction module 202: Defines neural implicit geometric features and appearance features in the local tangent space of the base mesh, and regresses the parameters of explicit three-dimensional Gaussian primitives from the features to construct a mesh-Gaussian mixture texture representation, thereby decoupling the mesoscopic structure texture from the base geometry;

[0143] Consistency Fine-tuning Sampling Module 203: Performs surface sampling on the mesh manifold, extracts neural texture blocks of mesoscopic structures, and implements consistency fine-tuning through sampling consistency loss and reconstruction consistency loss to reduce discrete sampling artifacts;

[0144] Neural texture synthesis module 204: applies smoothing regularization to the latent feature space and synthesizes high-resolution seamless neural textures from extracted neural texture blocks based on perceptual tilingability block matching and stitching.

[0145] Rendering module 205: Through tangent space mapping, it transfers the synthesized neural textures to the target 3D mesh and performs multi-texture compositing and real-time rendering.

[0146] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing.

[0147] Fourthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing as described above.

[0148] In addition, combined Figure 1 The method for extracting and synthesizing mesoscopic textures based on Gaussian splashing described in this application can be implemented by electronic devices, such as computer devices. Figure 9 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of this application.

[0149] In some embodiments, the computer device may further include a communication interface 83 and a bus 80. For example, Figure 9 As shown, the processor 81, memory 82, and communication interface 83 are connected through bus 80 and complete communication with each other.

[0150] Specifically, the processor 81 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0151] The memory 82 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 81.

[0152] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any of the mesoscopic structure texture extraction and synthesis methods based on Gaussian splashing in the above embodiments.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing, characterized in that, The method includes the following steps: Basic mesh extraction and preprocessing steps: Input the multi-view dataset of the acquired images and the preprocessed contour mask, and use the two-dimensional Gaussian splashing method to extract and optimize the two-dimensional basic mesh; obtain the multi-view image dataset containing camera poses and the preprocessed contour mask, and use the two-dimensional Gaussian splashing method to reconstruct and optimize the basic geometry to obtain a single-layer basic mesh. Mesh and Gaussian blend texture representation construction steps: Construct an embedded three-dimensional Gaussian splash of mesh and Gaussian blend texture representation in the tangent space of the two-dimensional basic mesh to construct the mesh-Gaussian blend texture representation and decouple the mesoscopic structure texture from the basic geometry; Consistency fine-tuning sampling steps: Surface sampling is performed on the mesh manifold to extract the neural texture blocks of the mesoscopic structure, and sampling artifacts caused by the mismatch between the sampling frequency and the actual density of the three-dimensional Gaussian are reduced by consistency fine-tuning including sampling consistency loss and reconstruction consistency loss; Neural texture synthesis steps: Apply smoothing regularization to the latent feature space and synthesize neural textures from the extracted neural texture blocks.

2. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing according to claim 1, characterized in that, The method further includes the following steps: Rendering steps: The synthesized neural texture is transferred to the target 3D mesh through tangent space mapping, and multi-texture compositing and real-time rendering are performed.

3. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing according to claim 1 or 2, characterized in that, The basic mesh extraction and preprocessing steps include: The input multi-view original object mask is filtered to generate a coarse-grained contour mask; Based on the contour mask, the basic geometry is reconstructed using 2D Gaussian splashing, with mask loss used for supervision during the reconstruction process. Based on various optimization coefficients, the mask loss, depth distortion loss, and normal consistency loss are jointly optimized. The reconstructed geometry is subjected to a visibility masking process. Rays are projected from the center of the training camera onto the mesh surface to remove invisible internal occlusion surfaces and redundant geometric structures, ultimately resulting in a single-layer 2D base mesh with adaptive subdivision.

4. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing according to claim 1 or 2, characterized in that, The mesh and Gaussian blend texture representation construction steps include: Neural implicit geometric features and appearance features are defined on the base mesh; the surface normals and texture features of the coordinates are interpolated using barycentric coordinates; The center coordinates of the 3D Gaussian are determined by predicting the offset of the Gaussian along the normal based on implicit geometric features using a shallow multilayer perceptron. The Gaussian center is encoded, and multiple sets of multilayer perceptrons are used to regress the Gaussian splash parameters, including opacity, scaling, rotation, and color; and the Gaussian direction and view direction are defined in the local tangent space of the base mesh surface.

5. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing according to claim 1 or 2, characterized in that, The consistency fine-tuning sampling step includes: Within the user-specified area of ​​the basic mesh manifold, uniformly sample local square tangent sampling planes and record the local mesh frame and the sampling plane frame; For each sampling plane, rays are projected from the pixels of the sampling plane along the surface normal direction and intersect the grid. Only when all rays intersect the grid within the specified area, the neural implicit texture features at the intersection point are queried by interpolation to form a locally structured neural texture block. The sampling consistency loss is used to constrain the rendering results of the sampled 3D Gaussian generated by the neural texture block and the reconstructed 3D Gaussian after consistency fine-tuning to be consistent in the sampling view and the training view. A reconstruction consistency loss is introduced to extract the original reconstruction quality before and after.

6. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing according to claim 1 or 2, characterized in that, The neural texture synthesis step includes: A smoothing regularization loss is constructed based on the difference in latent texture features between adjacent vertices in the base mesh to constrain the spatial continuity of latent texture features; When performing block matching and stitching, a block matching algorithm with perceptual tiling is adopted, and a differentiable tiling metric is introduced as a weight. For the stitching region and candidate texture blocks, a conditional probability distribution is constructed based on similarity, and seamless neural texture synthesis is completed by iterative sampling according to the conditional probability distribution.

7. The method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing as described in claim 2, characterized in that, The rendering steps include: For a target mesh with surface parameterization, surface points are randomly sampled, and the tangent space information of the synthetic texture is obtained through nearest neighbor interpolation in order to calculate the mapping transformation from the local tangent space of the source object to the local tangent space of the target 3D mesh through the tangent sampling plane. Based on the height factor, reference offset, and scaling factor, dynamically adjust the Gaussian center and scale on the target mesh; Real-time rendering is performed by performing differential rasterization on the updated Gaussian texture mapped to the target surface, and the height factor, reference offset, and scaling factor parameters are dynamically adjusted to achieve multi-texture composite generation.

8. A system for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing, characterized in that, The system employing the Gaussian splash-based mesoscopic structure texture extraction and synthesis method as described in any one of claims 1-7 includes the following modules: Basic mesh extraction and preprocessing module: acquires multi-view image datasets containing camera poses and preprocessed contour masks, reconstructs and optimizes basic geometry using two-dimensional Gaussian splashing, and obtains a single-layer basic mesh; Mesh and Gaussian Mixture Texture Representation Construction Module: Constructs an embedded 3D Gaussian splash of mesh and Gaussian mixture texture representation in the tangent space of the two-dimensional basic mesh to build a mesh-Gaussian mixture texture representation, decoupling the mesoscopic structure texture from the basic geometry; Consistency Fine-tuning Sampling Module: Performs surface sampling on the mesh manifold, extracts neural texture blocks of the mesoscopic structure, and reduces sampling artifacts caused by the mismatch between the sampling frequency and the actual density of the three-dimensional Gaussian through consistency fine-tuning including sampling consistency loss and reconstruction consistency loss; Neural texture synthesis module: applies smoothing regularization to the latent feature space and synthesizes neural textures from the extracted neural texture blocks; Rendering module: Through tangent space mapping, the synthesized neural texture is transferred to the target 3D mesh, and multi-texture compositing and real-time rendering are performed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for extracting and synthesizing mesoscopic structure textures based on Gaussian splashing as described in any one of claims 1-7.

Images