Three-dimensional grid vertex coloring method based on implicit representation and separated reflection color network

Abstract

The invention discloses a three-dimensional grid vertex coloring method, device and equipment based on an implicit representation and separated reflection color network. The method comprises the steps of collecting a multi-view data set of an object to be colored, estimating a camera pose through a multi-level view structure recovery method, and determining a standard sampling cubic domain and a grid file; training a NeuS framework to obtain a model containing a geometric branch network and a separated reflection color network; grid vertex coordinates and triangular surface indexes are extracted through a geometric branch network, a color network processes the vertex coordinates to obtain an RGB vertex set, and after gravity center interpolation smooth coloring is conducted, a color three-dimensional grid with explicit vertex colors is generated and written into a file; coloring is completed based on the file. According to the method, UV expansion and illumination modeling are not needed, accurate coloring is achieved through the separated color network, the color grid rendering efficiency is high, compatibility is high, and the problems of texture dislocation and the like can be effectively avoided.

Description

A Vertex Shading Method for 3D Mesh Based on Implicit Representation and Separate Reflection Color Network Technical Field

[0001] This application relates to the fields of 3D reconstruction, computer graphics and neural rendering technology, and in particular to a 3D mesh vertex shading method, apparatus and device based on implicit representation and a split reflective color network. Background Technology

[0002] 3D reconstruction methods based on implicit neural representation (INR) have attracted widespread attention in recent years, among which differentiable surface reconstruction frameworks, represented by NeuS, have advantages in reconstruction quality and geometric consistency. Unlike methods that directly predict dense voxels or point clouds, NeuS encodes the geometry and appearance of a scene as continuous implicit functions: the geometry is modeled by a signed distance function (SDF) with Eikonal constraints; the appearance is inferred by a color network based on the position of spatial points, normal vectors, and appearance features. This design allows the network to simultaneously optimize geometric consistency and volumetric rendering-based radiation consistency during training, resulting in a smoother and topologically sound implicit surface.

[0003] However, implicit representations cannot be directly used in traditional graphics pipelines, so in practical applications they usually need to be explicitly represented as polygonal meshes. NeuS uses the Marching Cubes algorithm to discretize the SDF on a 3D voxel mesh and extract vertices and triangular faces based on the zero level set. The resulting geometric mesh only contains spatial vertex coordinates and topological structure, still lacking texture information that can be used for rendering. Compared to density-volume-based models such as NeRF, NeuS's surface is an explicit 2D manifold, so querying the color of each mesh vertex is a natural and efficient way to generate textures: that is, using the network's own color rendering branch to predict a color consistent with the appearance of each mesh point in the training image.

[0004] While existing methods like NeuS can infer the signed distance field of an object from multi-view images and extract geometry using MarchingCubes, their native output typically only contains the geometric shape. To obtain a textured, renderable mesh, existing methods generally follow two paths. One is to use the traditional graphics workflow, performing UV unwrapping, lighting modeling, and texture optimization on the mesh. This type of method requires unwrapping the surface into a 2D texture space and generating texture maps through multi-view image optimization. However, UV unwrapping is prone to stretching, gaps, and overlaps in complex geometry, texture optimization requires long iterations, and lighting estimation and viewpoint differences further introduce errors, often resulting in high computational costs, texture distortion, or difficulty in accurately aligning colors with geometry. The other type of method does not use UVs but instead samples pixel colors frame-by-frame from the input image and projects them onto the mesh surface using light. This approach is simple in principle but requires setting multiple empirical thresholds to determine visibility, occlusion, and overlapping areas between multiple views, such as depth consistency thresholds, angle deviation thresholds, or occlusion confidence. These thresholds depend on empirical settings and require frequent parameter tuning for different scenarios. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, and device for 3D mesh vertex shading based on implicit representation and a separable reflective color network. The method utilizes the color network, SDF network, normal vector information, and multi-scale appearance features learned during implicit representation training to directly render the color of each vertex of a Marching Cubes mesh, thereby obtaining a vertex-shaded 3D model without UV mapping, lighting modeling, or texture gaps.

[0006] To achieve the above objectives, this application provides a 3D mesh vertex coloring method based on implicit representation and a separable reflection color network, comprising: acquiring a multi-view dataset of the object to be colored; estimating the camera pose in the multi-view dataset using a multi-level view structure recovery method; and determining the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset; training a pre-built NeuS framework based on the multi-view dataset to obtain a trained NeuS model, wherein the trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a separable reflection model network; extracting the mesh vertex coordinates and triangular face indices in the standard sampling cubic domain using the geometric branch network; processing the mesh vertex coordinates using the color network to obtain an RGB mesh vertex set; smoothly coloring the triangular facets according to the RGB color values ​​of the vertices using the centroid interpolation method to finally obtain a colored 3D mesh with explicit vertex colors; writing the colored 3D mesh with explicit vertex colors into a mesh file to obtain a colored mesh file; and coloring the object to be colored using the colored mesh file.

[0007] Optionally, the acquisition of the multi-view dataset includes: taking no fewer than thirty images of the object to be colored by uniformly distributing azimuth angles along the horizontal direction through multi-level viewpoints covering low, medium and high angles, and increasing the overlap rate between the shooting viewpoint and adjacent images when shooting complex objects to be colored.

[0008] Optionally, the step of estimating the camera pose in the multi-view dataset using a multi-level view structure recovery method and determining the standard sampled cubic domain and corresponding mesh file based on the camera pose estimation results includes: using the feature matching-based SFM method to estimate the extrinsic and intrinsic parameters of the camera for each photo to achieve optimal consistency between the camera trajectory, scene point cloud, and projection error; and normalizing each photo to normalize the shooting scene to a unit sphere or fixed bounding box to obtain the standard sampled cubic domain and corresponding mesh file.

[0009] Optionally, the step of extracting the grid vertex coordinates and triangular face indices in the standard sampling cube domain using a geometric branching network includes: based on the three-dimensional voxel grid within the standard sampling cube domain, sampling each voxel node using a geometric branching network, and then using the Marching Cubes algorithm to extract the zero level set of each sampling point to obtain a colorless geometric grid containing only the grid vertex coordinates and triangular face indices.

[0010] Optionally, the implicit feature vector is a multi-scale implicit feature generated within the geometric branch network; the step of processing the grid vertex coordinates using a color network to obtain an RGB grid vertex set includes: inputting the grid vertex coordinates into the geometric branch network to obtain the implicit feature vector and normal vector of the vertex; processing the normal vector and the incident direction in the camera pose based on the signed distance field gradient algorithm to obtain the reflection view direction; inputting the grid vertex coordinates, normal vector, implicit feature vector, and reflection view direction into the trained color network to output the BGR ordered color value of the vertex; and performing channel rearrangement on the BGR ordered color value to convert it into an RGB color value.

[0011] Optionally, before sampling each voxel node using a geometric branch network, the method further includes: projecting the pixels of each image into a ray based on the parameters of the input camera pose, and uniformly sampling each ray to obtain multiple sampling points.

[0012] Optionally, the split reflection model network includes a parallel diffuse reflection sub-network and a specular reflection sub-network. The two sub-networks are used to encode diffuse reflection components that are independent of the viewing direction and specular reflection components that are related to the viewing direction, respectively. The diffuse reflection sub-network and the specular reflection sub-network are the first multilayer perceptron and the second multilayer perceptron, respectively. The step of inputting the grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction into the trained color network and outputting the BGR order color value of the vertex includes: first, encoding the grid vertex coordinates through frequency position to enhance the high-frequency table, to obtain the encoded grid vertex coordinates; inputting the encoded grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction into the first multilayer perceptron, and outputting the diffuse reflection component; using viewpoint direction encoding, performing spherical harmonic encoding on the reflection viewing direction to obtain the spherically harmonic encoded reflection viewing direction; inputting the spherically harmonic encoded reflection viewing direction, normal vector, and grid vertex coordinates into the second multilayer perceptron, and outputting the specular reflection component; and performing feature concatenation of the diffuse reflection component and the specular reflection component, followed by linear mapping and Sigmoid activation to generate the BGR order color value of the vertex.

[0013] Optionally, determining the standard sampling cubic domain based on the multi-view dataset includes: selecting the pixel rays corresponding to the four corner points, four boundary midpoints, and the image center point of each training image in the multi-view dataset; extracting the near endpoints and far endpoints of each ray; updating the global minimum boundary and maximum boundary based on the near endpoints and far endpoints and adding outward padding; and normalizing the enclosing interval into a standard sampling cubic domain.

[0014] Furthermore, to achieve the above objectives, this application also provides a 3D mesh vertex coloring device based on implicit representation and a split-reflection color network, comprising: a data determination module, used to collect a multi-view dataset of the object to be colored, estimate the camera pose in the multi-view dataset using a multi-level view structure recovery method, and determine the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset; a model training module, used to train a pre-built NeuS framework based on the multi-view dataset to obtain a trained NeuS model, wherein the trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a split-reflection model network; a coloring file generation module, used to extract the mesh vertex coordinates and triangular face indices in the standard sampling cubic domain using the geometric branch network, process the mesh vertex coordinates using the color network to obtain an RGB mesh vertex set; smooth the triangular facets according to the RGB color values ​​of the vertices using the centroid interpolation method to finally obtain a colored 3D mesh with explicit vertex colors, and write the colored 3D mesh with explicit vertex colors into a mesh file to obtain a colored mesh file; and a coloring module, used to color the object to be colored using the colored mesh file.

[0015] To achieve the above objectives, this application also provides an electronic device, the electronic device comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the three-dimensional mesh vertex shading method based on implicit representation and split reflection color network provided in any of the foregoing embodiments.

[0016] This application proposes a method, apparatus, and device for 3D mesh vertex coloring based on implicit representation and a separable reflection color network. The method includes acquiring a multi-view dataset of the object to be colored; estimating the camera pose in the multi-view dataset using a multi-level view structure recovery method; determining the standard sampled cubic domain and the corresponding mesh file based on the multi-view dataset; training a pre-built NeuS framework based on the multi-view dataset to obtain a trained NeuS model, wherein the trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a separable reflection model network; extracting the mesh vertex coordinates and triangular face indices in the standard sampled cubic domain using the geometric branch network; processing the mesh vertex coordinates using the color network to obtain an RGB mesh vertex set; smoothly coloring the triangular faces according to the RGB color values ​​of the vertices using the centroid interpolation method to finally obtain a colored 3D mesh with explicit vertex colors; writing the colored 3D mesh with explicit vertex colors into a mesh file to obtain a colored mesh file; and coloring the object to be colored using the colored mesh file. This invention eliminates the reliance on empirical parameter control in the texture generation process, instead directly applying consistent color to the isosurface mesh based on color information obtained from implicit field rendering. Since manual threshold adjustment or viewpoint selection strategies are unnecessary, the entire texturing process can be automated, making it suitable for various data conditions and offering higher reproducibility and robustness. The mesh generated by this invention contains explicit vertex color information, eliminating the need to load additional texture maps to display the object's appearance. Compared to traditional methods relying on texture maps, vertex color display offers lower resource consumption and higher rendering efficiency, enabling smooth rotation, scaling, and real-time observation in common 3D viewing environments. Because the color is already bound to the mesh structure, it maintains good consistency during subsequent editing, deformation, or simplification, thereby improving the usability and flexibility of 3D representation. Attached Figure Description

[0017] Figure 1 is a flowchart of the process of extracting explicit meshes and generating consistent colors according to the present invention; Figure 2 is a color network improved based on the NeuS framework according to the present invention; Figure 3 is an SDF view of sampling points on a certain ray according to the present invention; Figure 4 is a color map of sampling points of a certain batch of rays according to the present invention; Figure 5 is a color map of sampling points of a certain batch of rays according to the present invention after multiplying by their weights; Figure 6 is a color map of surface sampling points retained after setting a threshold for a certain batch of rays according to the present invention; Figure 7 is the color mesh generation result of the present invention on public dataset 1; Figure 8 is the color mesh generation result of the present invention on public dataset 2; Figure 9 is the color mesh generation result of the present invention on self-built dataset 1; Figure 10 is the color mesh generation result of the present invention on self-built dataset 2; Figure 11 is the color mesh generation result of the present invention on self-built dataset 3.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0020] The main solution of this application embodiment is a 3D mesh vertex shading method based on implicit representation and a split reflective color network. Utilizing the color network, SDF network, normal vector information, and multi-scale appearance features learned during implicit representation training, the color of each vertex of the Marching Cubes mesh is directly rendered, thereby obtaining a vertex-shaded 3D model without UV mapping, lighting modeling, or texture gaps.

[0021] Referring to Figure 1, the three-dimensional mesh vertex coloring method based on implicit representation and split reflection color network provided in the first embodiment of this application may include: S101, collecting a multi-view dataset of the object to be colored, estimating the camera pose in the multi-view dataset through a multi-level view structure recovery method, and determining the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset; In one embodiment of this application, the process of collecting the multi-view dataset may include the following: by covering a multi-level view of low angle, medium angle and high angle, uniformly distributing the azimuth angle along the horizontal direction, taking no less than thirty images of the object to be colored, and increasing the overlap rate between the shooting view and adjacent images when shooting complex objects to be colored.

[0022] Multi-level view coverage ensures the model can fully observe the entirety of the object to be colored. Low-angle views capture bottom details, mid-angle views restore the main shape, and high-angle views supplement top features, avoiding the omission of geometric or color information due to missing viewpoints. Specifically, uniformly distributing azimuth angles along the horizontal direction ensures consistent viewpoint density in the horizontal direction, reducing situations where local viewpoints are too dense or sparse. For example, taking at least thirty images is a minimum requirement based on extensive experimental verification. This number provides sufficient multi-view constraints for model training, ensuring that implicit representations accurately encode scene information. For objects with high geometric complexity (such as complex cutouts or protruding structures) or rich details (such as fine textures or patterns on the surface), increasing the overlap rate between the shooting viewpoint and adjacent images can further improve multi-view consistency, helping the model learn the geometric and color relationships in details more accurately. It is understood that by specifying the specific requirements for dataset collection, this claim ensures that the collected images can provide comprehensive and consistent information support for subsequent model training, avoiding insufficient model training due to incomplete viewpoint coverage, insufficient quantity, or insufficient overlap, thereby ensuring the detail fidelity and overall consistency of the final colorization result.

[0023] In one embodiment of this application, the process of estimating the camera pose in a multi-view dataset using a multi-level view structure recovery method and determining the standard sampled cubic domain and the corresponding mesh file based on the camera pose estimation results may include the following: using the feature matching-based SFM method to estimate the extrinsic and intrinsic parameters of the camera for each photo to achieve optimal consistency between the camera trajectory, scene point cloud, and projection error; normalizing each photo to normalize the shooting scene to a unit sphere or fixed bounding box to obtain the standard sampled cubic domain and the corresponding mesh file.

[0024] Among them, the Feature Matching-based Structure-Focused Method (SFM) is a mature technique for camera pose estimation. It extracts and matches feature points between images to construct a sparse point cloud of the scene, and then solves for the camera extrinsic parameters (rotation, translation) and intrinsic parameters (focal length, principal point, etc.) of each image. This application uses the Feature Matching-based SFM method to estimate the camera extrinsic and intrinsic parameters of each image and adjusts the pose globally to achieve optimal consistency between camera trajectory, scene point cloud, and projection error. Simultaneously, the scene is scale-normalized so that the overall scene can be normalized to a unit sphere or fixed bounding box to ensure subsequent ray sampling density, neural network input scale, and optimization stability.

[0025] Specifically, the global pose adjustment process is achieved by minimizing projection error, ensuring that the deviation between the projected position of the sparse point cloud on each image and the actual feature point position is minimized, thereby guaranteeing the smoothness of the camera trajectory and the consistency of the scene point cloud. Optionally, scene normalization standardizes the captured scene to a unit sphere or fixed bounding box. Its purpose is to unify the input scale and avoid problems such as unreasonable ray sampling density and neural network input scale imbalance caused by differences in scene size, ensuring the stability of model training. The standard sampling cubic domain is determined based on the normalized scene range, providing a unified spatial benchmark for subsequent voxel sampling and mesh extraction. This step, through accurate camera pose estimation and standardized scene normalization, provides a reliable pose and spatial basis for subsequent model training and mesh processing. The accuracy of camera pose directly affects the accuracy of ray sampling and color projection, while scene normalization ensures the stability of model training and compatibility between different scenes, ultimately improving the accuracy of the colorization results.

[0026] The acquisition of multi-view datasets can be accomplished using conventional digital cameras or mobile devices. The core objective is to ensure comprehensive and uniform view coverage to support the accuracy of subsequent model training. Specifically, the shooting process must revolve around the object to be shaded, taking into account low, medium, and high viewpoints, and ensuring a uniform distribution of azimuth angles in the horizontal direction to avoid blind spots. It should be noted that camera pose estimation employs a feature-matching-based SFM (Structure-Factor-Based Modeling) method. This method can globally adjust the camera's extrinsic parameters (such as rotation matrix R and translation vector T) and intrinsic parameters (such as camera matrix K) to achieve optimal consistency between camera trajectory, scene point cloud, and projection errors, providing a reliable pose foundation for subsequent ray sampling and spatial modeling. Clearly, this step, through comprehensive dataset acquisition, accurate camera pose estimation, and standardized sampling domain determination, lays the data and spatial foundation for the entire shading process, ensuring that subsequent model training and mesh processing are based on realistic and consistent scene information, effectively reducing shading errors caused by data defects or unreasonable spatial ranges.

[0027] In one embodiment of this application, the process of determining the standard sampled cubic domain based on the multi-view dataset may include the following: selecting the pixel rays corresponding to the four corner points, four boundary midpoints, and the image center point of each training image in the multi-view dataset; extracting the near endpoints and far endpoints of each ray; updating the global minimum boundary and maximum boundary based on the near endpoints and far endpoints and adding outward padding; and normalizing the enclosing interval into a standard sampled cubic domain.

[0028] Nine representative pixels (four corner points, four boundary midpoints, and the image center point) of each image were selected because the rays corresponding to these pixels can cover the main field of view of the image, and their near and far endpoints can roughly delineate the spatial boundary of the scene from that perspective.

[0029] Specifically, after extracting the near and far endpoints of all rays, the global minimum and maximum boundaries are obtained by comparison to ensure that the sampling cubic domain can completely contain the entire scene.

[0030] Optionally, adding outward padding is to avoid the scene boundary coinciding with the cubic domain boundary, which would cause boundary clipping problems during subsequent voxel sampling or mesh extraction. The outward padding can be adaptively adjusted according to the scene size. Normalizing the enclosing interval into a standard sampling cubic domain is to take the maximum value of the absolute range of the three coordinate axes as a unified scale, construct a symmetrical cube, and ensure that the sampling density is consistent in all directions.

[0031] Clearly, this embodiment, through the selection of representative pixel rays, boundary calculation, and normalization processing, obtains a standard sampling cubic domain that can completely and uniformly cover the entire scene, providing a unified and reasonable spatial range for subsequent voxel sampling, mesh extraction, and vertex shading, effectively avoiding geometric omissions or shading deviations caused by insufficient or uneven spatial range.

[0032] S102. A pre-built NeuS framework is trained based on a multi-view dataset to obtain a trained NeuS model. The trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a separate reflection model network. The NeuS framework is a 3D reconstruction framework based on implicit neural representations. Its core advantage lies in encoding the geometry and appearance of the scene as continuous implicit functions, enabling simultaneous optimization of geometric consistency and radiation consistency. The geometric branch network is essentially an SDF (signed distance function) network. After inputting 3D spatial points, it outputs corresponding signed distance values ​​and a 256-dimensional implicit feature vector. This feature vector integrates the local structure of the points and the surface appearance information of the objects, providing crucial support for color prediction. For example, the color network employs a split reflection model design, which differs from traditional single-output color networks. It models diffuse and specular reflection components separately through two parallel sub-networks, enabling more accurate adaptation to the appearance characteristics of different materials and decoupling direction-dependent and direction-independent reflection components. It can be understood that this step, through training the model using the NeuS framework, achieves a deep binding between geometric and appearance information. In particular, the design of the split reflection color network provides algorithmic support for accurate vertex color prediction. Compared to traditional uniformly modeled color networks, it can more flexibly handle the viewpoint-dependent changes of complex materials, improving the physical consistency of color prediction.

[0033] In one embodiment of this application, the process of extracting the grid vertex coordinates and triangular face indices in the standard sampled cube domain using a geometric branching network may include the following: based on the three-dimensional voxel grid within the standard sampled cube domain, after sampling each voxel node using a geometric branching network, the zero level set of each sampling point is extracted using the Marching Cubes algorithm to obtain a colorless geometric grid containing only the grid vertex coordinates and triangular face indices.

[0034] The three-dimensional voxel mesh within the standard sampling cubic domain is a regular mesh constructed at a preset resolution (e.g., 64³, 128³), with each voxel node corresponding to a three-dimensional spatial location. Specifically, the geometric branch network samples each voxel node, essentially calculating the SDF value of that node to form a discrete approximation of the continuous SDF field. The SDF value reflects the distance from the node to the object surface and its internal / external relationship.

[0035] For example, the Marching Cubes algorithm is a classic algorithm for extracting implicit surfaces. Its core is to determine whether a voxel intersects the object's surface based on the SDF sign (positive / negative) of the voxel node. Then, it calculates the vertices on the zero-level set through interpolation, connects these vertices to form triangular patches, and finally obtains a colorless geometric mesh containing only spatial coordinates and topological structure. It can be understood that this step, through voxel sampling and the Marching Cubes algorithm, makes the implicit surface explicit. The resulting colorless geometric mesh is the basis for subsequent coloring processing. Its clear topology and complete structure ensure that the subsequent vertex colors accurately correspond to the object's surface positions, avoiding color misalignment or distortion caused by mesh structure defects.

[0036] In one embodiment of this application, the implicit feature vector is a multi-scale implicit feature generated within the geometric branch network; the process of using a color network to process the grid vertex coordinates to obtain an RGB grid vertex set may include the following: inputting the grid vertex coordinates into the geometric branch network to obtain the implicit feature vector and normal vector of the vertex; processing the normal vector and the incident direction in the camera pose based on the signed distance field gradient algorithm to obtain the reflection view direction; inputting the grid vertex coordinates, normal vector, implicit feature vector and reflection view direction into the trained color network to output the BGR ordered color value of the vertex; and performing channel rearrangement on the BGR ordered color value to convert it into an RGB color value.

[0037] Among them, the implicit feature vector is a 256-dimensional vector output by the geometric branch network (SDF network). It integrates the local structure of vertices and the appearance information of the object surface. It belongs to multi-scale implicit features and can provide rich contextual support for color prediction.

[0038] Specifically, the normal vector is calculated using the signed distance field gradient, reflecting the orientation of the object's surface at that vertex, and is a key parameter for constructing the reflection viewpoint direction. Specifically, the reflection viewpoint direction is constructed based on the signed distance field gradient (normal vector n) and the camera incident direction d, calculated using a specific formula. This direction ensures consistency between color prediction and the actual shading model, making the predicted color conform to the physical laws of lighting and viewing angle.

[0039] The specific formula is as follows: The expression for constructing the reflection observation direction based on the sign distance field gradient, i.e., the normal vector, is:

[0040] Where d is the incident direction of the camera; the expression for n is:

[0041] Ultimately It is fed into the color network as part of the input for training.

[0042] For example, the color network receives four types of inputs: vertex position, normal vector, implicit feature vector, and reflection viewing direction. It combines this information to perform color calculations, and its internal output color order is BGR. However, mainstream 3D files and display devices use the RGB format, so it is necessary to convert BGR to RGB through channel rearrangement to ensure correct color display.

[0043] In this embodiment, step S102 may include the following sub-steps for implicit modeling and multi-view training of the model.

[0044] Step S1021: Based on the input camera parameters, project the pixels of each training image into a ray r(x)=o+td.

[0045] Step S1022: Sample n=80 points uniformly for each ray.

[0046] Step S1023: Construct the SDF network. The x-coordinates of the sampling points are hashed and input into the network, outputting a 256-dimensional vector, where the first dimension is the predicted SDF value of that point. This 256-dimensional vector is then treated as a feature vector and used as input to the color network. As shown in Figure 3, the SDF values ​​of the sampling points of a randomly selected ray are visualized.

[0047] Step S1024: Construct the internal color network. Unlike the original NeuS framework, the network structure has been redesigned to accommodate the speed of hash encoding. As shown in Figure 2, a separate reflection model network is used to simultaneously predict diffuse and specular reflection components. This network uses spatial position x, surface normal vector n, and reflection viewing angle direction... The input consists of local features from the geometric network, which are encoded by two parallel sub-networks to encode orientation-dependent and orientation-independent reflection components, thereby generating the final pixel color.

[0048] Step S10241: The viewpoint direction vector is first encoded using a spherical harmonic function to obtain 16-dimensional directional features; the spatial coordinates are then mapped to a 24-dimensional high-frequency feature space through position encoding. Furthermore, the 256-dimensional local shape features provided by the geometric branch (SDF network) serve as additional input, offering rich geometric context support for appearance prediction.

[0049] Step S10242: The color network contains two symmetrical 256×256 MLP branches.

[0050] The first branch receives directional encoding and local features, which are used to predict viewpoint-dependent specular reflection terms:

[0051] in, This indicates the reflection direction used for color lookup. To improve the physical consistency of surface appearance modeling, this invention uses the following formula to calculate the reflection direction. It is expressed as:

[0052] The second branch relies solely on position-dependent input and is used to model the viewpoint-independent diffuse term:

[0053] The outputs of the two branches are concatenated at the feature level, then passed through a 64-dimensional linear mapping and activated by a sigmoid function to generate the final RGB color of the point.

[0054] This decoupled structure effectively decouples different reflective components, allowing the network to maintain its expressive power while improving its fitting stability, and enabling it to more accurately describe the viewpoint-dependent changes of complex materials. As shown in Figure 4, a batch of rays (512 rays) are visualized simultaneously, forming a fan shape.

[0055] Step S1025: Since each point has a weight, we multiply the α channel by these weights. The weight is higher when a ray passes through the scene surface. Setting a threshold for the weights yields the result shown in the figure. Further, merging several batches of rays yields the result shown in Figure 5. Figure 6 shows the approximate outline of a scene and the color of its surface.

[0056] This step achieves accurate prediction and standardization of vertex colors through the fusion of multi-dimensional inputs and standardized color format conversion. The introduction of implicit feature vectors improves the detail fidelity of color prediction, the construction of reflection view direction ensures the physical consistency of colors, and channel rearrangement ensures the compatibility of shading results, ultimately resulting in an accurate and usable set of RGB mesh vertices.

[0057] In one embodiment of this application, before sampling each voxel node using a geometric branch network, the method further includes: projecting each image pixel into a ray based on the parameters of the input camera pose, and uniformly sampling each ray to obtain multiple sampling points. The camera pose parameters include the extrinsic and intrinsic parameters estimated above. Projecting image pixels into rays based on these parameters essentially establishes a mapping relationship between image pixels and three-dimensional space, with each ray corresponding to the viewing direction of a pixel in the image.

[0058] Specifically, uniform sampling of each ray is performed to obtain a sufficient number of sampling points within the corresponding 3D space. These sampling points will serve as input to the Geometric Branch Network (SDF network) for learning the signed distance field of the scene. Optionally, the number of samples is typically set to 80 (adjustable according to scene complexity). Uniform sampling ensures a balanced distribution of sampling points along the ray direction, avoiding insufficient SDF field learning due to overly concentrated or sparse sampling points. In fact, this step, through ray casting and uniform sampling, provides sufficient training samples for the Geometric Branch Network, enabling the network to accurately learn the spatial distance information at different locations in the scene. This ensures the accuracy of subsequent voxel sampling and mesh extraction, providing a reliable geometric basis for vertex shading.

[0059] S103. A geometric branching network is used to extract the grid vertex coordinates and triangular face indices from the standard sampling cubic domain. A color network is used to process the grid vertex coordinates, resulting in an RGB grid vertex set. The triangular faces are then smoothly colored according to the RGB color values ​​of the vertices using centroid interpolation, ultimately yielding a colored 3D grid with explicit vertex colors. This colored 3D grid with explicit vertex colors is then written into a mesh file, resulting in a colored mesh file. The process of extracting the mesh using the geometric branch network is based on a 3D voxel mesh within a standard sampled cubic domain. After SDF sampling of each voxel node, the Marching Cubes algorithm is used to extract the zero level set (isosurface with SDF=0), resulting in a colorless geometric mesh containing vertex coordinates and triangular face indices. This mesh has a complete structure and clear topology, lacking only color attributes. Specifically, when processing vertex coordinates, the color network first inputs the vertex coordinates into the geometric branch network to obtain the corresponding normal vector and implicit feature vector. Then, it combines the constructed reflection viewpoint direction as input, calculates the diffuse and specular reflection components through a split sub-network, and finally outputs color values ​​in BGR order. After channel rearrangement, RGB color values ​​are obtained. Optionally, the application of the centroid interpolation method is to solve the problem of unnatural transitions between vertex and face colors. By interpolating the color of any point within a triangular face, the surface color presents a continuous and smooth distribution effect, avoiding abrupt color changes between vertices. To be precise, this step achieves the transformation from colorless mesh to colored mesh through the coherent processing of geometric extraction, color prediction and smooth shading. Moreover, the entire process does not rely on UV unwrapping or lighting modeling. It directly uses the information learned from implicit representation to complete the color assignment, which not only simplifies the process, but also avoids problems such as texture stretching, gaps or color alignment errors in traditional methods, and significantly improves the shading efficiency and consistency of the effect.

[0060] In one embodiment of this application, the split reflection model network includes a parallel diffuse reflection sub-network and a specular reflection sub-network. The two sub-networks are used to encode diffuse reflection components that are independent of the viewing direction and specular reflection components that are related to the viewing direction, respectively. The diffuse reflection sub-network and the specular reflection sub-network are the first multilayer perceptron and the second multilayer perceptron, respectively. The process of inputting the grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction into the trained color network and outputting the BGR order color value of the vertex may include the following: first, the grid vertex coordinates are frequency position encoded to enhance the high-frequency table, resulting in encoded grid vertex coordinates; the encoded grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction are input into the first multilayer perceptron, which outputs the diffuse reflection component; the reflection viewing direction is spherically harmonic encoded using viewpoint direction encoding, resulting in the spherically harmonic encoded reflection viewing direction; the spherically harmonic encoded reflection viewing direction, normal vector, and grid vertex coordinates are input into the second multilayer perceptron, which outputs the specular reflection component; the diffuse reflection component and the specular reflection component are feature concatenated, linearly mapped, and activated by Sigmoid to generate the BGR order color value of the vertex.

[0061] For example, steps S103 and S104 are used for geometric extraction based on the trained SDF network, and their sub-steps are as follows: Step S1031: First, determine the reconstruction range by setting the minimum / maximum boundary coordinates based on the bounding boxes of objects in the dataset. That is, the coordinates obtained in step S1032. scope.

[0062] Step S1032: Sample SDF values ​​on the 3D mesh at a certain resolution (e.g., 64³, 128³ or higher).

[0063] Step S1033: Use Marching Cubes to extract the mesh and obtain: vertex coordinates (N×3) and triangle indices (M×3).

[0064] Step S105 is based on the color network for mesh vertex coloring. The sub-steps of step S105 are as follows: Step S1051: For each vertex output by Marching Cubes, feed it into the SDF network to obtain the implicit geometric feature vector, and calculate the normal vector based on its gradient. If using automatic differentiation, disable gradient tracking during inference to save memory.

[0065] Step S1052: Since the mesh extraction process is directly performed through voxel sampling and does not involve the incident direction, we directly use the vertex normal vector as the viewing direction.

[0066] Step S1053: Combine the above quadruple (x,n,F, The input color network directly outputs the color of the vertex. The internal color order of the network is BGR, which needs to be rearranged through channels to obtain the standard RGB.

[0067] Step S1054: To accommodate different sizes of mesh vertex counts, the vertex set is divided into multiple batches and input into the network sequentially, thereby reducing resource consumption caused by a single input. That is, let the mesh vertex set be... We divide it into several subsets according to the predefined B:

[0068]

[0069] Step S1056: Facet Color Interpolation. For triangular faces, centroid interpolation is used to smoothly color the faces based on vertex colors, and the colors are written as vertex attributes into the mesh file. Figures 7–11 show the mesh coloring results obtained by the method of the present invention under public datasets and self-built datasets, respectively. As shown in the figures, in two types of data with different acquisition conditions, lighting environments, and texture complexities, the present invention can achieve stable, uniform, and detail-preserving vertex color regression after extracting the geometric structure.

[0070] Specifically, in publicly available datasets, due to the high image quality and uniform viewpoint distribution, the vertex colors generated by this invention are highly consistent with the real appearance, with clear texture boundaries and smooth color transitions. In self-built datasets, even with adverse factors such as changes in natural lighting, inconsistent camera equipment, and uneven viewpoint coverage, this invention can still effectively suppress the influence of noise and obtain mesh coloring results with continuous structure and consistent color, without obvious splicing marks, inconsistent exposure, or texture misalignment.

[0071] This step requires a separate network structure design and targeted feature encoding methods to enable the color network to more accurately model different reflective components. Compared with the traditional single network structure, it has higher fitting stability and can more accurately describe the appearance characteristics of complex materials. The final output BGR color value, after channel rearrangement, can obtain RGB colors that are highly consistent with the appearance of real objects.

[0072] S104. Use the colored mesh file to color the object to be colored.

[0073] The shaded mesh file contains explicit vertex color information, allowing it to be used directly for rendering and display without the need for additional texture mapping. Specifically, the file is compatible with common 3D viewing and editing software, supports smooth rotation, scaling, and real-time observation, and ensures that the colors maintain good consistency with the mesh structure during subsequent editing, deformation, or simplification, without texture misalignment or loss.

[0074] It's important to note that the core value of this step lies in transforming the colored 3D mesh obtained in the previous steps into a practically usable shaded result. Whether used for virtual demonstrations, product design, or other 3D applications, this mesh file can quickly and realistically reproduce the appearance of the object to be shaded. By directly applying the shaded mesh file, this step fully leverages the advantages of vertex colors. Compared to traditional shading methods that rely on texture mapping, it consumes fewer resources, has higher rendering efficiency, and is more compatible, thus fully demonstrating the practical value of the entire shading method.

[0075] Based on the above method embodiments, this application also provides a 3D mesh vertex coloring device based on implicit representation and a split reflection color network. The device may include: a data determination module, used to collect a multi-view dataset of the object to be colored, estimate the camera pose in the multi-view dataset using a multi-level view structure recovery method, and determine the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset; a model training module, used to train a pre-built NeuS framework based on the multi-view dataset to obtain a trained NeuS model, wherein the trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a split reflection model network; a coloring file generation module, used to extract the mesh vertex coordinates and triangular face indices in the standard sampling cubic domain using the geometric branch network, process the mesh vertex coordinates using the color network to obtain an RGB mesh vertex set; smooth the triangular facets according to the RGB color values ​​of the vertices using the centroid interpolation method to finally obtain a colored 3D mesh with explicit vertex colors, and write the colored 3D mesh with explicit vertex colors into a mesh file to obtain a colored mesh file; and a coloring module, used to color the object to be colored using the colored mesh file.

[0076] Based on the above method embodiments, this application also provides an electronic device, the electronic device comprising: at least one processor, a memory, and an input / output unit; wherein, the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the three-dimensional mesh vertex shading method based on implicit representation and split reflection color network provided in any of the foregoing embodiments.

[0077] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A three-dimensional mesh vertex coloring method based on implicit representation and a split reflective color network, characterized in that, include: Collect a multi-view dataset of the object to be colored, estimate the camera pose in the multi-view dataset using a multi-level view structure recovery method, and determine the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset; A pre-built NeuS framework is trained based on a multi-view dataset to obtain a trained NeuS model. The trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a split reflection model network. The geometric branch network is used to extract the grid vertex coordinates and triangular face indices in the standard sampled cubic domain, and the color network is used to process the grid vertex coordinates to obtain an RGB grid vertex set. The triangular facets are smoothly colored according to the RGB color values ​​of the vertices using the centroid interpolation method, and finally a colored 3D grid with explicit vertex colors is obtained. The colored 3D grid with explicit vertex colors is written into a grid file to obtain a colored grid file. Use the colored mesh file to color the object to be colored.

2. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The acquisition of the multi-view dataset includes: taking no fewer than thirty images of the object to be colored by uniformly distributing azimuth angles along the horizontal direction through multi-level viewpoints covering low, medium and high angles, and increasing the overlap rate between the shooting viewpoint and adjacent images when shooting complex objects to be colored.

3. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The method of estimating camera poses in a multi-view dataset using a multi-level view structure recovery method and determining the standard sampled cubic domain and corresponding mesh file based on the camera pose estimation results includes: using the feature matching-based SFM method to estimate the extrinsic and intrinsic parameters of the camera for each photo to achieve optimal consistency between camera trajectory, scene point cloud, and projection error; and normalizing each photo to normalize the shooting scene to a unit sphere or fixed bounding box to obtain the standard sampled cubic domain and corresponding mesh file.

4. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The step of extracting the grid vertex coordinates and triangular face indices in the standard sampling cube domain using a geometric branch network includes: based on the three-dimensional voxel grid within the standard sampling cube domain, sampling each voxel node using a geometric branch network, and then using the Marching Cubes algorithm to extract the zero level set of each sampling point to obtain a colorless geometric grid containing only the grid vertex coordinates and triangular face indices.

5. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The implicit feature vectors are multi-scale implicit features generated within the geometric branch network; The step of using a color network to process the grid vertex coordinates to obtain an RGB grid vertex set includes: inputting the grid vertex coordinates into a geometric branch network to obtain the implicit feature vector and normal vector of the vertex; The sign distance field gradient algorithm is used to process the normal vector and the incident direction in the camera pose to obtain the reflection view direction; the grid vertex coordinates, normal vector, implicit feature vector and reflection view direction are input into the trained color network to output the BGR order color value of the vertex; the BGR order color value is rearranged into channels and converted into RGB color value.

6. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, Before sampling each voxel node using a geometric branch network, the method further includes: projecting the pixels of each image into a ray based on the parameters of the input camera pose, and uniformly sampling each ray to obtain multiple sampling points.

7. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The split reflection model network includes a parallel diffuse reflection sub-network and a specular reflection sub-network. These two sub-networks are used to encode diffuse reflection components independent of the viewing direction and specular reflection components related to the viewing direction, respectively. The diffuse reflection sub-network and the specular reflection sub-network are the first and second multilayer perceptrons, respectively. The process of inputting the grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction into the trained color network and outputting the BGR order color value of the vertex includes: first, encoding the grid vertex coordinates through frequency position to enhance the high-frequency table, obtaining the encoded grid vertex coordinates; inputting the encoded grid vertex coordinates, normal vector, implicit feature vector, and reflection viewing direction into the first multilayer perceptron, outputting the diffuse reflection component; using viewpoint direction encoding, performing spherical harmonic encoding on the reflection viewing direction to obtain the spherically harmonic encoded reflection viewing direction; inputting the spherically harmonic encoded reflection viewing direction, normal vector, and grid vertex coordinates into the second multilayer perceptron, outputting the specular reflection component; and finally, concatenating the diffuse reflection component and the specular reflection component, applying a linear mapping and sigmoid activation to generate the BGR order color value of the vertex.

8. The three-dimensional mesh vertex coloring method based on implicit representation and split reflective color network as described in claim 1, characterized in that, The determination of the standard sampling cubic domain based on the multi-view dataset includes: selecting the pixel rays corresponding to the four corner points, four boundary midpoints, and the image center point of each training image in the multi-view dataset; extracting the near endpoints and far endpoints of each ray; updating the global minimum boundary and maximum boundary based on the near endpoints and far endpoints and adding outward padding; and normalizing the enclosing interval into a standard sampling cubic domain.

9. A three-dimensional mesh vertex shading device based on implicit representation and a split-reflection color network, characterized in that, include: The data determination module is used to collect multi-view datasets of the object to be colored, estimate the camera pose in the multi-view dataset through a multi-level view structure recovery method, and determine the standard sampling cubic domain and the corresponding mesh file based on the multi-view dataset. The model training module is used to train a pre-built NeuS framework based on a multi-view dataset to obtain a trained NeuS model. The trained NeuS model includes a geometric branch network and a color network connected in sequence, wherein the color network is a split reflection model network. The color file generation module is used to extract the grid vertex coordinates and triangular face indices in the standard sampled cubic domain using the geometric branch network, process the grid vertex coordinates using the color network, and obtain an RGB grid vertex set. The triangular facets are then smoothly colored according to the RGB color values ​​of the vertices using the centroid interpolation method, finally obtaining a colored 3D grid with explicit vertex colors. The colored 3D grid with explicit vertex colors is then written into a grid file to obtain a colored grid file. The coloring module is used to color the objects to be colored using the colored mesh file.

10. An electronic device, characterized in that, The electronic device includes: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the three-dimensional mesh vertex shading method based on implicit representation and split reflection color network according to any one of claims 1 to 8.

Images