A single-view three-dimensional texture consistency generation method based on geometric structure

Abstract

The application discloses a single-view three-dimensional texture consistency generation method based on geometric structure, pre-processes single-view image input data of a to-be-processed object and corresponding view angle parameters to obtain training input images, a view angle parameter set and supervision information, carries out geometric structure feature extraction and weight mapping processing on the training input images to obtain a geometric structure weight map, carries out sampling density distribution and position sampling on three-dimensional space texture query positions according to the geometric structure weight map to obtain a sampling point set, carries out multi-resolution coding on each sampling point in the sampling point set, and constructs a continuous texture representation through a multi-resolution texture hash grid to obtain texture attributes corresponding to each sampling point, and then carries out differentiable rendering, loss calculation and parameter updating to obtain a three-dimensional texture generation result. Compared with the prior art, the application improves the structure consistency, detail integrity and stability of three-dimensional texture generation under single-view conditions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D graphics technology, specifically relating to a single-view 3D texture consistency generation method based on geometric structure constraints. Background Technology

[0002] Neural radiation fields are a method for representing 3D scenes using continuous implicit functions. By mapping spatial coordinates and viewpoint direction to volume density and color information, and combining this with differentiable volume rendering, 3D structure reconstruction and texture generation are achieved. With the development of related technologies, 3D texture generation technology has been widely applied in fields such as digital content creation and 3D asset construction.

[0003] Existing 3D texture generation methods typically rely on multi-view image input. Under conditions of sufficient viewpoint information, existing methods can obtain relatively stable geometric structures and texture results; however, when only a single-view image is input, due to insufficient spatial constraints, problems such as structural distortion, pose drift, texture misalignment, and loss of detail are prone to occur, thus limiting the application of such methods in low-data-input scenarios.

[0004] To mitigate the impact of insufficient information from a single viewpoint, some existing methods introduce image priors or diffusion model distillation mechanisms to optimize the 3D generation process. While these methods can improve the generation results to some extent, their optimization process mainly relies on 2D image supervision signals and lacks explicit constraints on geometric structure information. This can easily lead to problems such as texture floating, blurred details, and inconsistent lighting in high curvature regions, boundary regions, and structurally complex regions. Meanwhile, existing methods typically employ uniform sampling or viewpoint-independent sampling during volume rendering, failing to allocate sampling density to 3D texture query locations based on geometric differences. This results in insufficient sampling in key areas, thus affecting the structural consistency and detail integrity of the generated 3D texture.

[0005] Therefore, existing technologies still have the following problems: insufficient structural consistency of 3D texture generation results under single-view conditions; lack of effective sampling density allocation mechanism for geometrically complex regions; and the need to improve the detail integrity and stability of the generated results. Summary of the Invention

[0006] The main objective of this invention is to overcome the shortcomings of existing technologies and provide a single-view 3D texture consistency generation method based on geometric structure. This method extracts geometric structure features and performs weight mapping on the training input image to obtain a geometric structure weight map. Based on this geometric structure weight map, it allocates sampling density and samples the location of texture queries in 3D space. Simultaneously, it encodes the sampling points using multi-resolution encoding and constructs a continuous texture representation using a multi-resolution texture hash grid. Combined with differentiable rendering, loss calculation, and parameter updates, it improves the consistency between geometric structure and texture representation under single-view conditions, thereby enhancing the structural consistency, detail integrity, and generation stability of the 3D texture generation results.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A single-view 3D texture consistency generation method based on geometric structure, the single-view 3D texture consistency generation method includes the following steps:

[0009] S1. Perform image size unification, pixel normalization, and foreground region extraction on the single-view image of the object to be processed and its corresponding camera view parameters to obtain the training input image I, the view parameter set Π, the effective image I*, and the supervision information Y used for consistency constraints. This step can reduce the size difference, pixel difference, and background interference between different input samples, and improve the consistency of subsequent geometric structure feature extraction, differentiable rendering, and loss calculation.

[0010] S2. After obtaining the training input image I, perform geometric structure feature extraction on it to extract the boundary information, contour direction information and local shape change information of the target object in the image, thereby forming geometric structure feature G. Geometric structure feature G is used to characterize the structural differences of the target object in the two-dimensional image.

[0011] S3. Perform weight mapping processing based on geometric structural features G to convert the degree of structural change at different locations of the target object into the corresponding weight distribution, and obtain the geometric structural weight map A. The geometric structural features G can be weighted according to the boundary response intensity, local shape change degree and detail transition degree of the target object at different locations to form a geometric structural weight map A that reflects the structural differences.

[0012] S4. Based on the weight values ​​at each location in the geometric structure weight map A, differentiated sampling density allocation is performed on the three-dimensional spatial texture query location, and location sampling is completed to obtain the sampling point set S. Among them, the area with a larger weight value corresponds to a higher sampling density, and the area with a smaller weight value corresponds to a lower sampling density, thereby making the sampling point distribution match the geometric structure distribution of the target object.

[0013] S5. Multi-resolution encoding is performed on each sampling point in the sampling point set S, and a texture representation is established in continuous space through a multi-resolution texture hash grid to obtain the texture attributes corresponding to each sampling point. The texture attributes include basic color, metallicity, roughness, and concavity / convexity normal vector. This step realizes the unified modeling of texture information at different scales, enabling sampling points to obtain texture attributes corresponding to their spatial positions.

[0014] S6. Perform differentiable rendering of the texture attributes corresponding to each sampling point according to the viewpoint parameter set Π. Under the imaging viewpoint determined by the viewpoint parameter set Π, map the texture attributes corresponding to each sampling point to the pixel color response in the image plane to obtain the rendering result R that is consistent with the viewpoint corresponding to the training input image I.

[0015] S7. Perform loss calculation on the rendering result R and the supervision information Y to obtain the loss value L, and update the parameters according to the loss value L so that the difference between the rendering result R and the training input image I gradually decreases. If the preset iteration end condition is not reached, continue to execute steps S5 to S6. When the preset iteration end condition is reached, the training is completed.

[0016] S8. After training is completed, forward rendering calculation is performed on the single-view image of the object to be processed based on the updated parameters to obtain a rendering result R that is consistent with the viewpoint of the single-view image of the object to be processed, and the rendering result R is determined as the three-dimensional texture generation result of the object to be processed.

[0017] Furthermore, the working process of S1 is as follows:

[0018] First, the original single-view image I0 of the object to be processed is scaled and normalized to obtain the training input image I; then, the foreground region is extracted from the training input image I, the foreground region Ω is determined according to the segmentation mask and the target detection box, and the training input image I is cropped to obtain the effective image I*, where the cropping process is expressed as: I*=Crop(I,Ω).

[0019] The foreground region extraction is used to obtain the target subject region and suppress background interference, so that the target subject region in the effective image I* remains intact and the background region has less impact on the subsequent texture generation process.

[0020] This technique addresses the issues of image size differences, pixel distribution differences, and background interference between different input images. Its design involves first scaling and normalizing the original single-view image I0 to unify the scale and pixel distribution of the input images. Then, foreground region extraction preserves the target subject region while suppressing background interference. The working principle is that by uniformly preprocessing the original single-view image I0, the training input image I and the effective image I* maintain consistency in spatial scale and pixel distribution, and the effective image I* more accurately reflects the image information of the target subject region. The technical effect is to improve the consistency and effectiveness of the training input image I and the effective image I*, and reduce the impact of background interference on subsequent geometric feature extraction, differentiable rendering, and loss calculation.

[0021] Furthermore, the working process of step S2 is as follows:

[0022] First, feature mapping is performed on the training input image I to obtain the basic feature representation F, expressed as follows: F = Ψ(I) * ;θ g Then, the gradient response is calculated on the basic feature representation F to obtain the feature gradient ∇F, as shown in the following expression: Then, based on the feature gradient ∇F, construct the structure intensity response map T(x, y) at pixel position (x, y), as shown in the following expression: Finally, the basic feature representation F is fused with the structural strength response map T(x, y) to obtain the geometric structural feature G, which is expressed as follows: G=Γ(F, T);

[0023] This technique addresses the challenge of reliably extracting boundary information, contour information, and local shape change information of target objects from single-view images. Its design involves first obtaining a basic feature representation F through feature mapping, then extracting structural change information through gradient response calculation, and further constructing a structural intensity response map T(x, y). Finally, the geometric structural features G are obtained by fusing the basic feature representation F and the structural intensity response map T(x, y). Its working principle is to use the basic feature representation F to characterize the initial geometric information in the image, and to enhance the response intensity of the target object's boundary region and local shape change region in the feature space using the feature gradient ∇F and the structural intensity response map T(x, y). The technical effect is to improve the accuracy and stability of geometric structural feature extraction.

[0024] Furthermore, the working process of step S3 is as follows:

[0025] First, the geometric structure feature G is weighted to obtain the geometric structure weight map A. Then, for any position p on the two-dimensional pixel plane where the geometric structure feature G is located, the Gaussian curvature k(p), normal change Δn(p), and edge saliency e(p) at that position are extracted. These three values ​​are then weighted and fused to obtain the weight value A(p) used to construct the geometric structure weight map A, as expressed below: ,

[0026] Finally, the weight value A(p) is compared with the preset threshold γ to obtain the set of geometrically significant regions M. sal ={p∣A(p)≥τ} and the set of geometrically insignificant regions M sal ={p∣A(p)<τ};

[0027] This technology addresses the problem of difficulty in quantifying and representing structural differences between different regions of a target object. Its design involves further mapping structural change information in the geometric feature G into position-related weight values ​​A(p), and then using a threshold to divide the weight values ​​A(p) into salient and insignificant geometric regions. Its working principle is based on using Gaussian curvature k(p), normal variation Δn(p), and edge saliency e(p) to jointly represent the degree of structural change of the target object at pixel position p, and then forming a geometric weight map A through normalization mapping. The technical effect is to improve the accuracy of geometric structural difference representation, enabling different regions to form weight distributions corresponding to their structural complexity.

[0028] Furthermore, the working process of step S4 is as follows:

[0029] First, the sampling density of the 3D spatial texture query position is allocated according to the weight value A(p) of the geometric structure weight map A at position p, and a sampling density function ρ(p) is constructed. The sampling density function ρ(p) satisfies a monotonic mapping relationship, and its expression is as follows: ;

[0030] Then, the location of the 3D spatial texture query position is sampled according to the sampling density function ρ(p), resulting in a set of sampling points S; finally, the k-th sampling point in the set of sampling points S is represented as p. k ;

[0031] This technology addresses the problem that uniform sampling methods struggle to simultaneously meet the sampling needs of both structurally complex and structurally smooth regions. Its design involves allocating differentiated sampling densities to the 3D spatial texture query locations based on the weight values ​​A(p) at different positions in the geometric structure weight map A, and then performing position sampling based on this density allocation. Its working principle is to establish a correspondence between the weight value A(p) and the sampling density through the sampling density function ρ(p), resulting in a denser distribution of sampling points in areas with larger weight values ​​A(p) and a sparser distribution in areas with smaller weight values ​​A(p). The technical effect is to improve the sampling accuracy in structurally complex regions and reduce redundant sampling in structurally smooth regions.

[0032] Furthermore, the working process of step S5 is as follows:

[0033] First, sample points p in the sample point set S. k Perform multi-resolution encoding to obtain sampling point p k The corresponding multi-resolution coding feature β(p) k ), multi-resolution coding feature β(p k ) is represented as: ,

[0034] Then the multi-resolution encoded feature β(p) k Input a multi-resolution texture hash mesh to obtain sampling points p k The corresponding texture properties are represented as follows:

[0035] ,

[0036] This technique addresses the problem that single-scale texture representation struggles to simultaneously capture global texture distribution and local detail variations; its design approach involves first processing the sampling points p... k Multi-resolution encoding is performed by stitching together texture information from different scale layers, and then the stitched multi-resolution encoded features are processed using a multi-resolution texture hash grid. Texture attribute mapping is performed; its working principle is to use multi-scale encoded features to jointly represent the sampling point p. k Texture information at different scales is used to establish sampling points p through a multi-resolution texture hash grid. k The correspondence between the base color, metallicity, roughness, and bump normal vectors; its technical effect lies in improving the ability of continuous texture representation to express multi-scale texture information, making the sampling point p k To obtain more accurate texture properties.

[0037] Furthermore, the working process of step S6 is as follows:

[0038] First, according to the viewpoint parameter set Π, each sampling point p is...k The corresponding texture attributes are rendered using differentiable methods to obtain a rendering result R that corresponds to the viewpoint of the training input image I. Then, a spatial ray is constructed at pixel position x in the image plane, denoted as r(t) = o + td. A sampling point p is then selected on the spatial ray r(t). k =r(t k Finally, based on each sampling point p... k The corresponding texture attributes are weighted and accumulated to obtain the rendering result R corresponding to pixel position x, which is represented as: ;

[0039] This technique addresses the problem that texture attributes in three-dimensional space are difficult to directly convert into a two-dimensional image with the same viewpoint as the training input image I. Its design involves first constructing a spatial ray in the image plane based on the viewpoint parameter set Π, and then selecting sampling points p along the spatial ray. k And for each sampling point p k The corresponding texture attributes are weighted and accumulated to form the rendering result R(x) at pixel position x. The rendering result R is composed of the rendering results R(x) at each pixel position x. Its working principle is: to establish the correspondence between the image plane and the three-dimensional space through spatial rays, and to accumulate and calculate the texture attributes on the spatial rays using volume rendering integrals. Its technical effect is to make the sampling point p k The corresponding texture attributes can form a stable two-dimensional rendering result R under the corresponding viewpoint.

[0040] Furthermore, the working process of the loss value L is as follows:

[0041] The loss value L is represented as: ;

[0042] This technique addresses the challenge of simultaneously achieving pixel consistency, optimization intensity for structurally significant regions, and texture attribute continuity under a single loss constraint; its design approach involves constructing a system based on the reconstruction consistency term L. rec Structural weighted constraint term L struct and spatial smoothing regularization term L reg The composite loss function, composed of these components, jointly constrains the rendering result R and the texture attributes corresponding to each sampling point; its working principle lies in utilizing the reconstruction consistency term L. rec The pixel difference between the constrained rendering result R and the training input image I is determined using the structure-weighted constraint term L. struct Enhance the parameter update intensity in regions with larger weight values ​​A(p) by utilizing the spatial smoothing regularization term L. reg It maintains the continuity of texture attributes in three-dimensional space; its technical effect is to improve the stability of the parameter update process and enhance the structural consistency and detail integrity of the three-dimensional texture generation results.

[0043] Furthermore, the working process of step S7 is as follows:

[0044] First, the gradients of the multi-resolution coding parameters and the multi-resolution texture hash grid parameters are calculated based on the loss value L; then, the parameters are iteratively updated using the backpropagation algorithm.

[0045] This technology addresses the problem that it is difficult to achieve effective collaborative optimization of multi-resolution encoding parameters and multi-resolution texture hash grid parameters solely through loss value constraints. Its design concept utilizes the loss value L as the basis for parameter updates, and establishes the gradient propagation relationship between the loss value L and the multi-resolution encoding parameters and multi-resolution texture hash grid parameters through a backpropagation algorithm. Its working principle involves calculating the gradients of the multi-resolution encoding parameters and multi-resolution texture hash grid parameters based on the loss value L, and then iteratively updating the parameters based on these gradients. The technical effect is to improve the stability of the parameter optimization process and enhance the accuracy of continuous texture representation.

[0046] Furthermore, the working process of step S8 is as follows:

[0047] First, the single-view image of the object to be processed is used as input to execute steps S1 to S6 to obtain a rendering result R with the same viewpoint. Then, the rendering result R is used as the three-dimensional texture generation result of the object to be processed. Finally, in the process of generating the three-dimensional texture result, only forward calculation is performed and no parameter update is performed.

[0048] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0049] 1. Improve the structural consistency of 3D texture generation results under single-view conditions. This invention extracts geometric structural features and performs weight mapping on the single-view image of the object to be processed to obtain a geometric structural weight map. Based on the geometric structural weight map, sampling density allocation and position sampling are performed on the 3D spatial texture query position, so that boundary areas, local shape change areas and detail transition areas receive a higher degree of sampling attention, thereby improving the structural consistency of the 3D texture generation results.

[0050] 2. Improve the ability to express texture details in key areas. Based on the weight values ​​of the geometric structure weight map at each location, this invention performs sampling density allocation and position sampling on the three-dimensional spatial texture query location. A denser distribution of sampling points is formed in areas with larger weight values, and a sparser distribution of sampling points is formed in areas with smaller weight values. This concentrates sampling resources on structurally complex areas, thereby improving the ability to express texture details in key areas and reducing redundant sampling in smooth areas.

[0051] 3. Improve the stability of continuous texture representation. This invention performs multi-resolution encoding on each sampling point in the sampling point set and constructs a continuous texture representation through a multi-resolution texture hash grid to obtain the texture attributes corresponding to each sampling point. This enables texture information at different scales to be uniformly expressed, thereby improving the stability of continuous texture representation and reducing discontinuities and distortions during texture generation.

[0052] 4. Improve the stability of the parameter optimization process. This invention performs differentiable rendering on the texture attributes corresponding to each sampling point according to the viewpoint parameter set, obtaining a rendering result consistent with the viewpoint corresponding to the training input image. It also combines supervised information for loss calculation and parameter update, so that a stable consistency constraint is formed between the rendering result and the training input image, thereby improving the stability of the parameter optimization process and the consistency of the 3D texture generation result. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of the single-view three-dimensional texture consistency generation method based on geometric structure of the present invention, which shows the processing flow from single-view image input to three-dimensional texture generation result output;

[0055] Figure 2 This is a schematic diagram of geometric structure weight graph generation and sampling density allocation disclosed in this invention, which includes geometric structure feature extraction, geometric structure weight graph A generation, sampling density allocation ρ(p), position sampling, and sampling point set S;

[0056] Figure 3 This is a schematic diagram of the construction and differentiable rendering of multi-resolution continuous texture representation disclosed in this invention, which includes a multi-scale coding layer, multi-scale feature stitching, multi-resolution texture hash grid and differentiable rendering. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0058] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0059] Example 1

[0060] This embodiment discloses a single-view 3D texture consistency generation method based on geometric structure, which specifically includes the following steps:

[0061] S1. Preprocessing and constructing supervision information for the object to be processed:

[0062] The single-view image of the object to be processed and its corresponding camera viewpoint parameters are processed by image resizing, pixel normalization, and foreground region extraction to obtain the training input image I and the viewpoint parameter set Π={K, R, t}. The training input image I is then cropped to obtain the effective image I. * And form supervision information Y for consistency constraints, wherein the supervision information Y includes valid image I. * The pixel color observation information and the alignment information required for viewpoint consistency constraints;

[0063] In this embodiment, all input images are uniformly adjusted to a resolution of 512×512 and scaled using bilinear interpolation; the RGB three-channel pixel values ​​are normalized to the [0,1] interval; to reduce background interference, target region localization and foreground region extraction are performed on the input images, so that the target subject area occupies about 75% of the image area and the target is located in the center of the image; the camera parameter information corresponding to the single-view image input data is parsed to obtain the view parameter set Π; based on the training input image I and the corresponding view parameter set Π, the supervision information Y for consistency constraints is constructed.

[0064] S2. Extract geometric structure features:

[0065] First, feature mapping is performed on the training input image I to obtain the basic feature representation F: F = Ψ(I) * ;θ g Then, the gradient response is calculated on the basic feature representation F to obtain the feature gradient ∇F: Then, construct the structural intensity response map T(x, y) at pixel position (x, y) based on the feature gradient ∇F: Finally, the basic feature representation F is fused with the structural strength response map T(x, y) to obtain the geometric structural feature G: G=Γ(F, T);

[0066] In this embodiment, a three-layer convolutional structure is used to extract features from the training input image I. The kernel size of each layer is 3×3, and the number of output feature channels is 32. The structural boundary, contour direction and local shape change information in the training input image I are extracted through the three-layer convolutional structure to obtain geometric structural features G, and the feature dimension of the geometric structural features G is 32.

[0067] S3. Weight mapping processing and construction of geometric structure weight graph:

[0068] First, the geometric structure feature G is weighted to obtain the geometric structure weight map A. Then, for any position p on the two-dimensional pixel plane where the geometric structure feature G is located, the Gaussian curvature k(p), normal change Δn(p), and edge saliency e(p) at that position are extracted and weighted to obtain the weight value A(p) used to construct the geometric structure weight map A. Its expression is as follows:

[0069]

[0070] Finally, the weight value A(p) is compared with the preset threshold γ to obtain the set of geometrically significant regions M. sal ={p∣A(p)τ} and the set of geometrically insignificant regions M sal ={p∣A(p)<τ};

[0071] In this embodiment, a 1×1 convolution layer is used to perform feature mapping on the geometric structure feature G, and a geometric structure weight map A is obtained through the Sigmoid activation function; the weight value of the geometric structure weight map A at position p is represented as A(p);

[0072] The positions in the training input image I are divided by comparing the weight value A(p) with the threshold τ. In this embodiment, the threshold τ is 0.6. When A(p)≥0.6, the corresponding position is determined to be a geometrically significant region, and the other positions are determined to be non-significant regions. The division results of geometrically significant and non-significant regions are used for subsequent sampling density allocation.

[0073] S4. Sampling density allocation and location sampling:

[0074] First, the sampling density of the 3D spatial texture query position is allocated according to the weight value A(p) of the geometric structure weight map A at position p, and a sampling density function ρ(p) is constructed. The sampling density function ρ(p) satisfies a monotonic mapping relationship: ;

[0075] Then, the location of the 3D spatial texture query position is sampled according to the sampling density function ρ(p), resulting in a set of sampling points S; finally, the k-th sampling point in the set of sampling points S is represented as p.k ;

[0076] Combination such as Figure 2 The diagram shown illustrates the generation of the geometric structure weight map and the allocation of sampling density. The number of basic sampling points for each ray is 64, which is increased to 96 in the salient areas of the structure, in order to achieve a distribution strategy of denser sampling in salient areas and relatively sparse sampling in non-salient areas.

[0077] S5, Multi-resolution Encoding and Texture Construction:

[0078] First, sample points p in the sample point set S. k Perform multi-resolution encoding to obtain sampling point p k The corresponding multi-resolution coding feature β(p) k ): ,

[0079] Then the multi-resolution encoded feature β(p) k Input a multi-resolution texture hash mesh to obtain sampling points p k The corresponding texture properties are represented as follows: ,

[0080] In this embodiment, as Figure 3 The construction of the multi-resolution continuous texture representation shown includes three scale branches: low-scale, medium-scale, and high-scale. The three scale branches contain 5-layer, 5-layer, and 6-layer coding structures, respectively, for a total of 16 coding layer structures, which are used to construct continuous texture representations at different resolutions.

[0081] Each sampling point in the sampling point set S is encoded using multi-resolution methods through three scale branches to obtain three-layer encoded features with feature dimensions of 32, 64, and 128 respectively. The three-layer encoded features are then concatenated using multi-scale methods to obtain a 224-dimensional feature vector. This 224-dimensional feature vector is then input into a multi-resolution texture hash grid for feature lookup and mapping to obtain the texture attributes corresponding to each sampling point.

[0082] S6, Differentiable rendering:

[0083] First, according to the viewpoint parameter set Π, each sampling point p is... k The corresponding texture attributes are rendered using differentiable methods to obtain a rendering result R that corresponds to the viewpoint of the training input image I. Then, a spatial ray is constructed at pixel position x in the image plane, denoted as r(t) = o + td. A sampling point p is then selected on the spatial ray r(t). k =r(t k Finally, based on each sampling point p... k The corresponding texture attributes are weighted and accumulated to obtain the rendering result R corresponding to pixel position x: ;

[0084] In this embodiment, spatial rays corresponding to pixel positions are constructed in the image plane according to the viewpoint parameter set Π, and the texture attributes corresponding to each sampling point are weighted and accumulated based on the volume rendering model to generate a rendering result R; the rendering result R is consistently aligned with the training input image I under the corresponding viewpoint.

[0085] In this embodiment, the depth parameter corresponding to the near clipping plane is set to 0.1, and the depth parameter corresponding to the far clipping plane is set to 6.0; the number of sampling points on each spatial ray is adaptively adjusted to 64 to 96 according to the weight value of the geometric structure weight map A at the corresponding position.

[0086] S7. Loss Calculation and Parameter Update:

[0087] Firstly, based on the loss value Calculate the gradients of the multi-resolution encoding parameters and the multi-resolution texture hash grid parameters; then use the backpropagation algorithm to iteratively update the parameters.

[0088] Loss calculation is performed on the rendered image R and the supervision information Y to obtain the loss value L, and the parameters are updated according to the loss value L. If the preset iteration end condition is not met, steps S5 to S6 are continued; if the preset iteration end condition is met, the training is completed.

[0089] In this embodiment, the Adam optimizer is used to update the parameters in the single-view 3D texture consistency generation method. The initial learning rate is set to 1e-4, the number of training rounds is set to 100, and the batch size is set to 8.

[0090] S8. Texture generation result output:

[0091] First, the single-view image of the object to be processed is used as input to execute steps S1 to S6 to obtain a rendering result R with the same viewpoint; then, the rendering result R is used as the 3D texture generation result of the object to be processed; finally, in the process of generating the 3D texture generation result, only forward calculation is performed and no parameter update is performed.

[0092] In this embodiment, a single-view image of the object to be processed is input into a trained single-view 3D texture consistency generation method, and forward calculation is performed based on the trained generation parameters to obtain the 3D texture generation result of the object to be processed.

[0093] The data used in the single-view image experiment of the object to be processed is the Objaverse dataset proposed by Deitke et al. in the paper "Objaverse: A Universe of Annotated 3D Objects". This dataset contains more than 800,000 3D objects with geometric structure and semantic annotation information, covering multiple object types, and providing mesh models and material information in a unified format.

[0094] The entire network training process is as follows: Figure 1 As shown. After network training is completed, refer to... Figure 1 The test process involves taking the single-view input image to be processed as input, and performing structure extraction and texture reconstruction processing through a consistency generation network to obtain the three-dimensional texture generation result.

[0095] Table 1 presents a quantitative comparison of the proposed method with currently popular 3D texture generation methods FlashTex, FlexiTex, and MVPaint on the Objaverse dataset test set. Evaluation metrics include SSIM, LPIPS, Chamfer Distance, Normal Consistency, and FID. Higher values ​​for SSIM and Normal Consistency are better, while lower values ​​for LPIPS, Chamfer Distance, and FID are better. The table shows that the proposed method outperforms existing methods in SSIM, LPIPS, Chamfer Distance, and Normal Consistency, demonstrating better performance in structural consistency and texture detail restoration. Furthermore, it achieves superior results in FID, indicating that the 3D textures generated by the proposed method have advantages in overall perceptual quality.

[0096] Table 1. Comparison of the results of the three-dimensional texture consistency generation method for geometric structures disclosed in this invention with existing methods.

[0097]

[0098] Example 2

[0099] This embodiment discloses a single-view 3D texture consistency generation method based on geometric structure, which specifically includes the following steps:

[0100] S1. Preprocessing and constructing supervision information for the object to be processed:

[0101] The single-view image of the object to be processed and its corresponding camera viewpoint parameters are processed by image resizing, pixel normalization, and foreground region extraction to obtain the training input image I and the viewpoint parameter set Π={K, R, t}. The training input image I is then cropped to obtain the effective image I. * And form supervision information Y for consistency constraints, wherein the supervision information Y includes valid image I. * The pixel color observation information and the alignment information required for viewpoint consistency constraints;

[0102] In this embodiment, all input images are uniformly adjusted to a resolution of 640×640 and scaled using bilinear interpolation; the RGB three-channel pixel values ​​are normalized to the [0,1] interval; to reduce background interference, target region localization and foreground region extraction are performed on the input images, so that the target subject area occupies about 75% of the image area and the target is located in the center of the image; the camera parameter information corresponding to the single-view image input data is parsed to obtain the view parameter set Π; based on the training input image I and the corresponding view parameter set Π, the supervision information Y for consistency constraints is constructed.

[0103] S2. Extract geometric structure features:

[0104] First, feature mapping is performed on the training input image I to obtain the basic feature representation F: F = Ψ(I) * ;θ g Then, the gradient response is calculated on the basic feature representation F to obtain the feature gradient ∇F: Then, construct the structural intensity response map T(x, y) at pixel position (x, y) based on the feature gradient ∇F: Finally, the basic feature representation F is fused with the structural strength response map T(x, y) to obtain the geometric structural feature G: G=Γ(F, T);

[0105] In this embodiment, a three-layer convolutional structure is used to extract features from the training input image I. The kernel size of each layer is 3×3, and the number of output feature channels is 64. The structural boundary, contour direction and local shape change information in the training input image I are extracted through the three-layer convolutional structure to obtain geometric structural features G, and the feature dimension of the geometric structural features G is 64.

[0106] S3. Weight mapping processing and construction of geometric structure weight graph:

[0107] First, the geometric structure feature G is weighted to obtain the geometric structure weight map A. Then, for any position p on the two-dimensional pixel plane where the geometric structure feature G is located, the Gaussian curvature k(p), normal change Δn(p), and edge saliency e(p) at that position are extracted and weighted to obtain the weight value A(p) used to construct the geometric structure weight map A. Its expression is as follows: ,

[0108] Finally, the weight value A(p) is compared with the preset threshold γ to obtain the set of geometrically significant regions M. sal ={p∣A(p)≥τ} and the set of geometrically insignificant regions M sal ={p∣A(p)<τ};

[0109] In this embodiment, a 1×1 convolution layer is used to perform feature mapping on the geometric structure feature G, and a geometric structure weight map A is obtained through the Sigmoid activation function; the weight value of the geometric structure weight map A at position p is represented as A(p), where p represents the position in the training input image I;

[0110] The positions in the training input image I are divided by comparing the weight value A(p) with the threshold τ. In this embodiment, the threshold τ is 0.5. When A(p)≥0.5, the corresponding position is determined to be a geometrically significant region, and the other positions are determined to be non-significant regions. The division results of geometrically significant and non-significant regions are used for subsequent sampling density allocation.

[0111] S4. Sampling density allocation and location sampling:

[0112] First, the sampling density of the 3D spatial texture query position is allocated according to the weight value A(p) of the geometric structure weight map A at position p, and a sampling density function ρ(p) is constructed. The sampling density function ρ(p) satisfies a monotonic mapping relationship: ;

[0113] Then, the location of the 3D spatial texture query position is sampled according to the sampling density function ρ(p), resulting in a set of sampling points S; finally, the k-th sampling point in the set of sampling points S is represented as p. k ;

[0114] Combination such as Figure 2 The diagram shown illustrates the generation of the geometric structure weight map and the allocation of sampling density. The number of basic sampling points for each ray is 96, which is increased to 128 in the salient areas of the structure, in order to achieve a distribution strategy of denser sampling in salient areas and relatively sparse sampling in non-salient areas.

[0115] S5, Multi-resolution Encoding and Texture Construction:

[0116] First, sample points p in the sample point set S. k Perform multi-resolution encoding to obtain sampling point p k The corresponding multi-resolution coding feature β(p) k ): ,

[0117] Then the multi-resolution encoded feature β(p) k Input a multi-resolution texture hash mesh to obtain sampling points p k The corresponding texture properties are represented as follows: ,

[0118] In this embodiment, as Figure 3 The construction of the multi-resolution continuous texture representation shown includes three scale branches: low-scale, medium-scale, and high-scale. The three scale branches contain 6-layer, 6-layer, and 6-layer coding structures, respectively, for a total of 18 coding layer structures, which are used to construct continuous texture representations at different resolutions.

[0119] Each sampling point in the sampling point set S is encoded using the three scale branches to obtain three-layer encoded features with feature dimensions of 64, 128, and 256 respectively. The three-layer encoded features are then concatenated using multi-scale features to obtain a 448-dimensional feature vector. The 448-dimensional feature vector is then input into a multi-resolution texture hash grid for feature query and mapping to obtain the texture attributes corresponding to each sampling point.

[0120] S6, Differentiable rendering:

[0121] First, according to the viewpoint parameter set Π, each sampling point p is... k The corresponding texture attributes are rendered using differentiable methods to obtain a rendering result R that corresponds to the viewpoint of the training input image I. Then, a spatial ray is constructed at pixel position x in the image plane, denoted as r(t) = o + td. A sampling point p is then selected on the spatial ray r(t). k =r(t k Finally, based on each sampling point p... k The corresponding texture attributes are weighted and accumulated to obtain the rendering result R corresponding to pixel position x: ;

[0122] In this embodiment, spatial rays corresponding to pixel positions are constructed in the image plane according to the viewpoint parameter set Π, and the texture attributes corresponding to each sampling point are weighted and accumulated based on the volume rendering model to generate a rendering result R; the rendering result R is consistently aligned with the training input image I under the corresponding viewpoint.

[0123] In this embodiment, the depth parameter corresponding to the near clipping plane is set to 0.1, and the depth parameter corresponding to the far clipping plane is set to 6.0; the number of sampling points on each spatial ray is adaptively adjusted to 96 to 128 according to the weight value of the geometric structure weight map A at the corresponding position.

[0124] S7. Loss Calculation and Parameter Update:

[0125] Firstly, based on the loss value Calculate the gradients of the multi-resolution encoding parameters and the multi-resolution texture hash grid parameters; then use the backpropagation algorithm to iteratively update the parameters.

[0126] Loss calculation is performed on the rendered image R and the supervision information Y to obtain the loss value L, and the parameters are updated according to the loss value L. If the preset iteration end condition is not met, steps S5 to S6 are continued; if the preset iteration end condition is met, the training is completed.

[0127] In this embodiment, the Adam optimizer is used to update the parameters in the single-view 3D texture consistency generation method. The initial learning rate is set to 1e-4, the number of training rounds is set to 200, and the batch size is set to 8.

[0128] S8. Texture generation result output:

[0129] First, the single-view image of the object to be processed is used as input to execute steps S1 to S6 to obtain a rendering result R with the same viewpoint; then, the rendering result R is used as the 3D texture generation result of the object to be processed; finally, in the process of generating the 3D texture generation result, only forward calculation is performed and no parameter update is performed.

[0130] In this embodiment, a single-view image of the object to be processed is input into a trained single-view 3D texture consistency generation method, and forward calculation is performed based on the trained generation parameters to obtain the 3D texture generation result of the object to be processed.

[0131] The data used in the single-view image experiment of the object to be processed is the Objaverse dataset proposed by Deitke et al. in the paper "Objaverse: A Universe of Annotated 3D Objects". This dataset contains more than 800,000 3D objects with geometric structure and semantic annotation information, covering multiple object types, and providing mesh models and material information in a unified format.

[0132] Table 2 shows the quantitative comparison results of the present invention with the currently popular 3D texture generation methods FlashTex, FlexiTex, and MVPaint on the Objaverse dataset test set. The evaluation metrics include SSIM, LPIPS, Chamfer Distance, Normal Consistency, and FID. Higher values ​​for SSIM and Normal Consistency are better, while lower values ​​for LPIPS, Chamfer Distance, and FID are better. The table shows that the present invention outperforms existing methods in SSIM, LPIPS, Chamfer Distance, and Normal Consistency, demonstrating better performance in structural consistency and texture detail restoration. Furthermore, it also achieves superior results in FID, indicating that the 3D textures generated by the present invention have advantages in overall perceptual quality.

[0133] Table 2 Comparison of the results of the three-dimensional texture consistency generation method for geometric structures disclosed in this invention with existing methods.

[0134]

[0135] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0136] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for generating consistent 3D textures from a single perspective based on geometric structure, characterized in that, The single-view 3D texture consistency generation method includes the following steps: S1. Perform image resizing, pixel normalization, and foreground region extraction on the single-view image of the object to be processed and its corresponding camera view parameters to obtain the training input image I and the view parameter set Π={K, R, t}, where K represents the camera intrinsic parameter matrix, R represents the camera extrinsic parameter rotation matrix, and t represents the camera extrinsic parameter translation vector. Crop the training input image I to obtain the effective image I. * And form supervision information Y for consistency constraints, wherein the supervision information Y includes valid image I. * The pixel color observation information and the alignment information required for viewpoint consistency constraints; S2. Extract geometric structure features from the training input image I to obtain geometric structure features G that characterize the boundary, contour direction, and local shape changes of the target object; S3. Perform weight mapping on the geometric structure features G to obtain the geometric structure weight map A for the sampling density allocation of the three-dimensional spatial texture query position; S4. Based on the weight values ​​of each position in the geometric structure weight map A, the three-dimensional spatial texture query position is sampled for density allocation and position sampling to obtain the sampling point set S; S5. Perform multi-resolution encoding on each sampling point in the sampling point set S, and construct a continuous texture representation through a multi-resolution texture hash grid to obtain the texture attributes corresponding to each sampling point. The texture attributes of the sampling points include basic color, metallicity, roughness, and bump normal vector texture attributes. S6. Perform differentiable rendering of the texture attributes corresponding to each sampling point according to the viewpoint parameter set Π to obtain the rendering result R that is consistent with the viewpoint corresponding to the training input image I; S7. Calculate the loss between the rendered image R and the supervision information Y to obtain the loss value L, and update the parameters according to the loss value L. If the preset iteration end condition is not met, continue to execute steps S5 to S6; if the preset iteration end condition is met, the training is completed. S8. Take the single-view image of the object to be processed as input and execute steps S1 to S6 to obtain a rendering result R that corresponds to the single-view image of the object to be processed. Use the rendering result R as the three-dimensional texture generation result of the object to be processed.

2. The method for generating consistent three-dimensional textures based on geometric structure according to claim 1, characterized in that, The process of step S1 is as follows: The original single-view image I0 of the object to be processed is scaled and pixel normalized to obtain the training input image I; Foreground region extraction is performed on the training input image I to obtain the target subject region and suppress background interference. The foreground region Ω is determined based on the segmentation mask and the target detection box. The training input image I is then cropped to obtain the effective image I. * The cutting process is represented as I * =Crop(I,Ω), where Crop(,) is the cropping operation.

3. The method for generating consistent three-dimensional textures based on geometric structure according to claim 1, characterized in that, The process of step S2 is as follows: S21. Perform feature mapping on the training input image I to obtain the basic feature representation F, expressed as follows: F=Ψ(I * ;θ g ), where Ψ(⋅) represents the feature mapping function, θ g Indicates the feature mapping parameters; S22. Calculate the gradient response of the basic feature representation F to obtain the feature gradient ∇F, as shown in the following expression: ; in, The basic feature represents the partial derivative of F with respect to the pixel's horizontal coordinate x. The basic feature represents the partial derivative of F with respect to the pixel's vertical coordinate y; S23. Construct the structure intensity response map T(x,y) at pixel position (x,y) based on the feature gradient ∇F, with the following expression: ,in Represents the L2 norm operation; S24. The basic feature representation F is fused with the structural strength response map T(x,y) to enhance the response intensity of the target object boundary region and local shape change region in the feature space, resulting in the geometric structural feature G=Γ(F, T); where Γ(·) represents the feature fusion function.

4. The method for generating consistent three-dimensional textures based on geometric structure according to claim 1, characterized in that, The process of step S3 is as follows: S31. Perform weight mapping on the geometric structure features G to obtain the geometric structure weight map A; S32. For any position p on the two-dimensional pixel plane where the geometric structure feature G is located, extract the Gaussian curvature k(p), normal change Δn(p), and edge saliency e(p) at that position, perform weighted fusion, and calculate the weight value A(p) used to construct the geometric structure weight map A. Its expression is: , Where α, β, and γ represent the weighting coefficients corresponding to Gaussian curvature k(p), normal variation Δn(p), and marginal significance e(p), respectively, and Softmax(·) represents the normalized mapping function; S33. Compare the weight value A(p) with the preset threshold γ to obtain the set of geometrically significant regions M. sal ={p∣A(p)≥τ} and the set of geometrically insignificant regions M sal ={p∣A(p)<τ}, where γ represents the threshold used to divide the geometrically significant and non-significant regions.

5. The method for generating consistent three-dimensional textures based on geometric structure according to claim 1, characterized in that, The process of step S4 is as follows: S41. Based on the weight value A(p) of the geometric structure weight map A at position p, the sampling density of the three-dimensional spatial texture query position is allocated, and the sampling density function ρ(p) is constructed. The sampling density function ρ(p) must satisfy a monotonic mapping relationship, expressed as: dq represents the integral variable in the continuous spatial domain M, and η represents the density adjustment coefficient. Represents the normalization term over the continuous spatial domain M; S42. Based on the sampling density function ρ(p), perform position sampling on the three-dimensional spatial texture query position to obtain the sampling point set S; S43. Represent the k-th sampling point in the sampling point set S as p. k k represents the sampling point number; a denser sampling point distribution is formed in areas with larger weight values ​​A(p), and a sparser sampling point distribution is formed in areas with smaller weight values ​​A(p).

6. The method for generating consistent three-dimensional textures based on geometric structure according to claim 5, characterized in that, The process of step S5 is as follows: S51. For sampling points p in the sampling point set S k Perform multi-resolution encoding to obtain sampling point p k The corresponding multi-resolution coding feature β(p) k ), multi-resolution coding feature β(p k ) is represented as: , in, Indicates sampling point p k The encoded features at the l-th scale layer, l=1,2,…,L; where L represents the total number of scale layers, and Concat(·) represents the feature concatenation operation; S53, Encode the multi-resolution feature β(p) k Input a multi-resolution texture hash mesh to obtain sampling points p k The corresponding texture properties are represented as follows: ， Where Γ(·) represents the texture attribute mapping operation function of the multi-resolution texture hash mesh, and k c Indicates sampling point p k The corresponding base color, k m Indicates sampling point p k The corresponding metallicity, k r Indicates sampling point p k The corresponding roughness, k n Indicates sampling point p k The corresponding concave / convex normal vector.

7. The method for generating consistent three-dimensional textures based on geometric structure according to claim 6, characterized in that, The process of step S6 is as follows: S61. According to the viewpoint parameter set Π, sample points p are... k The corresponding texture attributes are rendered in a differentiable manner to obtain a rendering result R that is consistent with the viewpoint corresponding to the training input image I; S62. Construct the corresponding spatial ray at pixel position x in the image plane, represented as: r(t)=o+td, where r(t) represents the spatial ray corresponding to pixel position x, o represents the position of the camera optical center corresponding to the imaging viewpoint, d represents the ray direction vector determined by pixel position x and the viewpoint parameter set Π, and t represents the depth parameter along the spatial ray. S63. Select a sampling point p on the spatial ray r(t). k =r(t k );in, This represents the depth parameter corresponding to the kth sampling point, where k represents the sampling point number; S64, Based on each sampling point p k The corresponding texture attributes are weighted and accumulated to obtain the rendering result R corresponding to pixel position x, which is represented as: , Among them, t n With t f Let represent the near clipping plane and the far clipping plane of the ray, respectively, and T(r(t)) represent the color response corresponding to the position on the spatial ray r(t). This represents the volume density function value corresponding to the position on the spatial ray r(t).

8. The method for generating consistent 3D textures from a single perspective based on geometric structure according to claim 7, characterized in that, The loss value L is represented as: , in , , These are the weighting coefficients of the reconstruction consistency term, the structural weighted constraint term, and the spatial smoothing regularization term, respectively. rec This represents the reconstruction consistency term, used to constrain the pixel consistency between the rendered result R and the training input image I at the corresponding viewpoint. L struct This represents a structural weighting constraint term that assigns higher update weights to regions with larger weight values ​​A(p); L reg This represents a spatial smoothing regularization term, used to constrain the continuity of texture attributes corresponding to each sampling point in three-dimensional space.

9. The method for generating consistent three-dimensional textures based on geometric structure according to claim 7, characterized in that, In step S7, the parameters are updated based on the loss value L. The updated parameters include multi-resolution encoding parameters and multi-resolution texture hash grid parameters. The specific rule for updating the parameters is as follows: based on the loss value L, the gradient of the above parameters is calculated through the backpropagation algorithm and iteratively updated.

10. The method for generating consistent three-dimensional textures based on geometric structure according to claim 1, characterized in that, The process of step S8 is as follows: S81. Take the single-view image of the object to be processed as input and execute steps S1 to S6 to obtain the rendering result R with the same viewpoint. S82. Use the rendering result R as the 3D texture generation result of the object to be processed; S83. In the process of generating the three-dimensional texture, only forward calculation is performed, and no parameter updates are performed.

Images