A method and system for generating 3D assets based on geometric conditions

Abstract

The application relates to the technical field of image processing, and discloses a 3D asset generation method and system based on geometric conditions, which comprises the following steps: generating a three-dimensional geometry based on a generation condition feature, and uniformly arranging a plurality of camera perspectives around the three-dimensional geometry, and synchronously rendering based on the same camera parameters; synchronously projecting the three-dimensional geometry based on the camera parameters and the pixel projection relationship of each camera perspective, and determining a position map and a normal map; a multi-perspective texture diffusion model taking the position map and the normal map as double geometric conditions to perform texture diffusion processing, and uniformly constraining the three-dimensional surface points in different perspectives to determine the texture images of the camera perspectives; projecting the texture images of the camera perspectives to the same UV texture space, fusing the textures projected to the same UV pixels, and determining the 3D asset of the target UV texture map. The application ensures the reliability of 3D asset generation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for generating 3D assets based on geometric conditions. Background Technology

[0002] In applications such as virtual reality, augmented reality, game development, digital content creation, and industrial simulation, 3D assets typically include three-dimensional geometry describing the spatial shape and surface structure of an object, and texture maps describing the object's surface color, pattern, and material details. With the development of generative artificial intelligence and diffusion models, automatically generating 3D geometry based on text or image prompts, and further generating textures based on that geometry, has become an important way to improve the efficiency of 3D asset production.

[0003] Existing patent application CN119131299A discloses a method, device, storage medium, and program product for generating 3D model textures guided by stylized images. This method obtains the stylized texture of the 3D object by acquiring a mesh model of the 3D object and a stylized image for guidance, and then performing UV sampling based on an optimized texture representation model. However, in multi-view texture generation scenarios for 3D assets, when the geometric sources of images with different geometric conditions are inconsistent, the spatial position and surface normal direction represented by the same pixel cannot accurately correspond. This easily leads to texture attachment position offsets or local orientation inconsistencies on the 3D surface. Furthermore, when texture images from different camera perspectives are further projected onto the same UV texture space, multiple perspectives provide different texture colors to the same UV pixel, resulting in texture conflicts and insufficient reliability in 3D asset generation.

[0004] Therefore, it is necessary to design a 3D asset generation method and system based on geometric conditions to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a 3D asset generation method and system based on geometric conditions, which aims to solve the problems that when the geometric sources of images with different geometric conditions are inconsistent, the textures on the three-dimensional surface are prone to attachment position shifts or local orientation inconsistencies, and when the texture images from different camera perspectives are further projected onto the same UV texture space, texture conflicts will occur, resulting in insufficient reliability of 3D asset generation.

[0006] This invention proposes a 3D asset generation method based on geometric conditions, comprising: Receive and encode text or image prompts input by the user, determine generation condition features, generate three-dimensional geometry based on the generation condition features, and uniformly set several camera viewpoints around the three-dimensional geometry, and perform synchronous rendering based on the same camera parameters. The three-dimensional geometry is synchronously projected based on the camera parameters and pixel projection relationship of each camera viewpoint to determine the position map and normal map. The position map records the three-dimensional spatial position of each pixel, and the normal map records the surface normal direction of each pixel. The generation condition features, the position map and normal map of each camera view, and the camera parameters are input into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and the normal map as dual geometric conditions to perform texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different views to determine the texture image of each camera view. The three-dimensional geometry is UV unwrapped, the texture images from each camera viewpoint are projected onto the same UV texture space, and the textures projected onto the same UV pixels are fused to determine the initial UV texture map. The seam areas of the initial UV texture map are corrected to determine the 3D asset of the target UV texture map.

[0007] Furthermore, when determining the generation condition features, the following are included: The text prompt is input into a pre-trained text encoder to extract the text semantics of the text prompt and encode them into text conditional features. The image prompt is input into a pre-trained image encoder to extract the image content, structure and style information of the image prompt and encode them into image conditional features. The generated conditional features include the text conditional features and the image conditional features.

[0008] Furthermore, when generating 3D geometry based on generation conditional features, it includes: The 3D geometry generation model based on the VAE-DiT architecture denoises the generation condition features and generates truncated symbolic distance field volume data, and determines the three-dimensional geometry based on isosurface extraction. The resolution of the truncated symbol distance from the field volume is 128. 3 Or 256 3 The cutoff distance is 0.1-0.2 times the bounding box radius.

[0009] Furthermore, when determining the location map and normal map, the following should be included: Configure camera intrinsics and extrinsic parameters, and use the same camera intrinsics, camera extrinsic parameters, and rasterization rendering pipeline for each camera viewpoint to generate depth maps, visibility masks, surface point index maps, position maps, and normal maps; The location map and the normal map are derived from the same 3D geometry, camera viewpoint, and pixel projection relationship, and the same pixels in the location map and the normal map correspond to the same 3D surface point; The depth map is used to record the depth value of each pixel; The visibility mask is used to mark whether each pixel is visible; The surface point index map is used to record the patch number and centroid coordinates corresponding to each pixel.

[0010] Furthermore, in determining the texture images for each camera viewpoint, the following is included: The text conditional features, the position maps and normal maps of each camera viewpoint, the camera intrinsic parameters and the camera extrinsic parameters are input into the multi-view texture diffusion model, which is constructed based on the SDXL diffusion model architecture. The location map is used to constrain the three-dimensional spatial position of the texture, and the normal map is used to constrain the local orientation and texture direction. In each denoising time step of the multi-view texture diffusion model, the same three-dimensional surface points from different camera views are constrained to be consistent.

[0011] Furthermore, in determining the texture images for each camera viewpoint, the following is also included: The consistency constraints include one or more combinations of constraint algorithms: The constraint algorithm includes feature aggregation, cross attention, shared noise, and consistency loss; The feature aggregation then averages or weighted averages the features of the same three-dimensional surface points from different camera perspectives. The cross-attention method uses features from one camera viewpoint as the Query and features from other camera viewpoints as the key. It then performs a dot product calculation based on the Query and the key to determine the similarity, and performs a weighted summation of the values ​​based on the similarity.

[0012] Furthermore, in determining the texture images for each camera viewpoint, the following is also included: The shared noise generates the same noise for all camera views of the same three-dimensional surface point; The consistency loss extracts the features of 3D surface points under different camera views and forms feature pairs. It calculates the squared difference between each feature pair, accumulates all squared differences, and backpropagates the accumulated result together with the loss of the multi-view texture diffusion model.

[0013] Furthermore, during texture blending, the following are included: If several camera viewpoints correspond to the same UV pixels, then a fusion weight is calculated. The fusion weight is the product of the viewpoint angle weight, the normal direction weight, the visibility weight, and the confidence level. The confidence level is the uncertainty estimate of the multi-view texture diffusion model. The initial UV texture map is determined based on the fusion weight.

[0014] Furthermore, when correcting the seam areas of the initial UV texture map, the following steps are included: Determine the UV island boundaries, occlusion areas, and conflict areas in the initial UV texture map; Based on the correspondence of the same three-dimensional surface points, texture correction is performed on the UV island boundary, the occlusion area and the conflict area from the perspective of adjacent camera or adjacent UV region using one or more of the following methods: texture dilation, neighborhood completion, Poisson fusion and diffusion repair.

[0015] Compared with existing technologies, the advantages of this invention are as follows: By receiving text or image prompts and extracting generation condition features, and then setting several camera perspectives around the same 3D geometry, a unified and stable geometric foundation is provided for subsequent texture generation. By synchronously generating position maps and normal maps under the same camera parameters and the same pixel projection relationship, the two types of geometric conditions correspond to the same 3D surface point at the pixel level, avoiding alignment errors caused by differences in geometric sources, rendering steps, or projection parameters. Specifically, the position map is used to define the spatial attachment position of the texture on the 3D surface, and the normal map is used to define the local surface orientation and texture extension direction. Inputting both as dual geometric conditions into the multi-view texture diffusion model can improve the accuracy and directional consistency of texture positioning on complex curved surfaces, edges, and uneven areas. By applying consistency constraints to the features of the same 3D surface point under different camera perspectives, the colors, patterns, and material boundaries generated from different perspectives remain coordinated, reducing color abrupt changes, detail drift, and texture breaks that occur when switching perspectives. After projecting texture images from various camera perspectives into the same UV texture space, weighted fusion is performed based on the viewpoint angle, normal direction, visibility, and confidence level to reduce the impact of occlusion, oblique views, and low-confidence perspectives on the final texture map. UV island boundaries, occluded areas, and conflict areas are corrected through texture dilation, neighborhood completion, Poisson fusion, or diffusion repair, improving the continuity and integrity of the target UV texture map. This forms a complete processing flow from 3D geometry generation, dual geometry condition construction, multi-view texture diffusion, consistency constraints to UV fusion and seam correction, improving the automation level, texture stability, and visual quality of 3D asset generation.

[0016] On the other hand, this application also provides a geometry-based 3D asset generation system for applying the above-mentioned geometry-based 3D asset generation method, including: The geometry generation module is configured to receive and encode text or image prompts input by the user, determine generation condition features, generate three-dimensional geometry based on the generation condition features, and uniformly set several camera viewpoints around the three-dimensional geometry, and perform synchronous rendering based on the same camera parameters. The synchronous generation module is configured to synchronously project the three-dimensional geometry based on the camera parameters and pixel projection relationship of each camera viewpoint to determine the position map and normal map. The position map records the three-dimensional spatial position corresponding to each pixel, and the normal map records the surface normal direction corresponding to each pixel. The condition construction module is configured to input the generated condition features, the position map and normal map of each camera view, and camera parameters into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and the normal map as dual geometric conditions to perform texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different views to determine the texture image of each camera view. The texture diffusion module is configured to perform UV unwrapping on the three-dimensional geometry, project texture images from each camera viewpoint onto the same UV texture space, fuse textures projected onto the same UV pixels, determine an initial UV texture map, correct the seam areas of the initial UV texture map, and determine the 3D asset of the target UV texture map.

[0017] It is understandable that the above-mentioned geometrically based 3D asset generation method and system have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a geometrically condition-based 3D asset generation method provided in an embodiment of the present invention; Figure 2 This is a functional block diagram of a geometrically based 3D asset generation system provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] See Figure 1 As shown in some embodiments of this application, a 3D asset generation method based on geometric conditions includes: S100: Receives and encodes text or image prompts input by the user, determines the generation condition features, generates 3D geometry based on the generation condition features, and uniformly sets several camera viewpoints around the 3D geometry, and performs synchronous rendering based on the same camera parameters. S200: Based on the camera parameters and pixel projection relationship of each camera view, the three-dimensional geometry is synchronously projected to determine the position map and normal map. The position map records the three-dimensional spatial position of each pixel, and the normal map records the surface normal direction of each pixel. S300: The generated condition features, position maps and normal maps of each camera viewpoint, and camera parameters are input into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and normal map as dual geometric conditions for texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different viewpoints to determine the texture image of each camera viewpoint. S400: Performs UV unwrapping on 3D geometry, projects texture images from each camera viewpoint onto the same UV texture space, merges textures projected onto the same UV pixels, determines the initial UV texture map, corrects the seam areas of the initial UV texture map, and determines the 3D asset of the target UV texture map.

[0023] Specifically, in S100, text prompts are input from the user describing the object category, structural features, appearance, material, or texture style of the 3D asset to be generated in natural language. For example, a text prompt might describe an object as a mechanical part, furniture, or decoration, and describe its surface as having a metal, wood, or fabric texture. Image prompts are two-dimensional images used to provide a reference for the content, structure, or style of the object to be generated. During 3D asset generation, one or both text prompts and image prompts can be received, or both can be received simultaneously. When only text prompts are received, only text encoding is performed; when only image prompts are received, only image encoding is performed. When both text and image prompts are received simultaneously, text conditional features and image conditional features are generated separately.

[0024] When determining text conditional features, text prompts are input into a pre-trained text encoder. The text encoder (such as CLIP, T5, etc.) encodes the natural language description into a high-dimensional feature vector, which serves as the conditional input to the generative model. The text encoder can encode object names, shape descriptions, component relationships, and material descriptions in the text. Object names are used to limit the type of subject to be generated, shape descriptions and component relationships affect 3D geometry, and material descriptions and style descriptions affect subsequent texture images. When determining image conditional features, image prompts are input into a pre-trained image encoder. The image encoder extracts the subject contours, local structures, color distribution, and style information from the image and transforms them into image conditional features. These generated conditional features are not directly used as the final geometry or texture, but rather as conditional inputs in the subsequent model denoising process, causing the random latent variables to gradually converge towards the 3D geometry and texture results that conform to the user prompts.

[0025] It should be noted that the text encoder and image encoder in this application are preferably implemented using existing technologies in the field, and are not the focus of the improvement claimed in this application. The focus of the improvement in this application lies in determining the data connection relationship and processing logic of the generation condition features. Those skilled in the art can complete the adaptation, replacement, or equivalent implementation based on the input-output relationship disclosed in this application, combined with existing disclosed technologies or conventional engineering methods, without affecting the implementation of the technical solution of this application.

[0026] After the generation condition features are determined, they are input into a 3D geometry generation model based on the VAE-DiT architecture, which includes a variational autoencoder (VAE) and a diffusion transformer (DiT). The variational autoencoder is used to represent 3D volume data in a latent space, reducing the computational cost of directly processing high-resolution 3D voxel data. The diffusion transformer performs multi-step denoising on the latent variables based on the generation condition features, gradually restoring the latent variables to truncated signed distance field volume data representing the shape of the object to be generated. The truncated signed distance field is a compact and efficient 3D geometry representation that implicitly represents 3D geometry by storing the signed distance from each point in the storage space to the object surface. Positive distances are represented on the outside, and negative distances are represented on the inside. It also supports efficient voxel fusion and mesh reconstruction operations, enabling the model to focus on learning geometric changes close to the object surface. After the 3D geometry generation model is denoised, TSDF (Truncated Signed Distance Field) volume data is output. Isosurface extraction is performed on the truncated signed distance field volume data through marching cubes. The object surface is determined at the position where the distance value is zero or close to zero, generating a 3D mesh composed of vertices and faces, and this 3D mesh is determined as the 3D geometry.

[0027] The 3D geometry can also be determined using other geometry generation or reconstruction methods. The resolution of the truncation symbol distance field volume is set to 128 cubed or 256 cubed. 128 cubed is suitable for objects with simpler contours and an emphasis on generation speed, while 256 cubed is suitable for objects with more grooves, corners, and surface details. The truncation distance is set to 0.1-0.2 times the bounding box radius. The model range is represented by a cuboid bounding box that completely encloses the 3D geometry. The bounding box radius is the distance from the center of the axis-aligned bounding box of the 3D geometry to the vertex of the bounding box, which is half the length of the diagonal of the bounding box space.

[0028] Isosurface extraction transforms volumetric data into a mesh format suitable for rasterization rendering, UV unwrapping, and texture application. Subsequent location maps, normal maps, depth maps, visibility masks, and surface point index maps all use this 3D geometry as their data source, eliminating the need to call different geometry versions separately and thus avoiding misalignment of condition maps due to inconsistencies in geometry versions. The center of the bounding box of the 3D geometry is used as the observation center for the camera's viewpoint, and several camera viewpoints are evenly distributed around the 3D geometry. Eight or twelve surround viewpoints can be used, with adjacent camera viewpoints maintaining the same angular interval.

[0029] For objects whose top structure needs to participate in texture generation, a top viewpoint can be set based on the surrounding viewpoint. The more cameras there are, the more comprehensive the coverage of the object's surface, but the more condition maps and texture images need to be generated and processed. For objects with regular structures and minimal occlusion, eight views can be used; for objects with complex surfaces or significant mutual occlusion between parts, twelve views can be used. The camera distance can be set according to the bounding box radius, ensuring that the 3D geometry is completely within the image range and occupies the main imaging area. Each camera viewpoint is configured with camera intrinsic and extrinsic parameters. The camera intrinsic parameters describe projection parameters such as focal length, principal point position, and image size, while the camera extrinsic parameters describe the position and orientation of the current camera viewpoint relative to the 3D geometry.

[0030] In S200, the same 3D geometry is input into the rasterization rendering pipeline for each camera viewpoint. Based on camera intrinsic and extrinsic parameters, the rasterization rendering pipeline projects the faces in the 3D mesh onto the 2D image plane, determines the pixels covered by each face, and interpolates the corresponding 3D surface information based on the pixel's position within the face. Under the same projection process or the same set of projection parameters, the 3D spatial position, surface normal direction, depth, visibility status, face number, and centroid coordinates are written to the corresponding image channels, thereby synchronously generating a position map, normal map, depth map, visibility mask, and surface point index map.

[0031] The position map records the 3D spatial position of each valid pixel. For a given pixel, the rasterization rendering pipeline first determines the triangular facet into which the pixel falls. Then, based on the centroid coordinates of the pixel within the triangular facet, it interpolates the spatial coordinates of the three vertices of the facet to obtain the 3D coordinates of the corresponding surface point. To facilitate processing by multi-view texture diffusion models, the 3D coordinates are normalized based on the bounding box center and bounding box size of the 3D geometry. Thus, the position map can directly provide the position of the corresponding surface point in 3D space for each pixel.

[0032] The normal map records the surface normal direction corresponding to the same pixel in the position map. The rasterization pipeline interpolates and normalizes the normal vector of a 3D surface point based on the vertex normals or facet normals of the triangular facets, using the same centroid coordinates. Since the position map and normal map are calculated using the same facets, centroid coordinates, and pixel positions, the 3D spatial position recorded by a pixel in the position map and the surface normal direction recorded by the same pixel in the normal map belong to the same 3D surface point.

[0033] The depth map records the depth values ​​of 3D surface points relative to the current camera viewpoint and is used to help determine the occlusion relationship. The visibility mask marks whether a pixel corresponds to a surface that is directly visible from the current viewpoint. The surface point index map records the facet and centroid coordinates corresponding to the pixel, enabling 2D pixels to trace back to the specific surface position on the 3D mesh.

[0034] When simultaneously generating these condition maps (depth map, visibility mask, surface point index map, position map, and normal map), the image size and pixel coordinate system are kept consistent; for example, each condition map uses a resolution of 512×512 or 1024×1024. If the position map uses one resolution and the normal map uses another, even if they originate from the same 3D geometry, pixel correspondence deviations will occur due to resampling. Therefore, this embodiment uses the same resolution and the same cropping range under the same camera viewpoint, and writes each condition map separately from the same rasterization result. This ensures that the position map and normal map are homogeneous, synchronized, and aligned at the pixel level, avoiding errors introduced by subsequent registration of separately generated condition maps.

[0035] Based on surface point index maps and visibility masks, the correspondence between the same 3D surface points under different camera viewpoints can be determined. Specifically, the patch number and barycentric coordinates are combined to form the 3D surface point identifier. The patch number represents the triangular patch in which each 3D surface point is located; the encoding of the patch number can be set manually and will not be described in detail. The barycentric coordinates are used to determine the position of the surface point inside the triangular patch. When a 3D surface point is visible under multiple camera viewpoints, the pixel coordinates of the 3D surface point identifier in each camera viewpoint are recorded, thus forming a correspondence between the same 3D surface point and pixels in multiple viewpoints.

[0036] In the S300, the generated conditional features, position maps and normal maps for each camera viewpoint, camera intrinsics, and camera extrinsics are input into the multi-view texture diffusion model. The multi-view texture diffusion model is built on the SDXL (Stable Diffusion XL) diffusion model architecture, capable of generating texture images at a resolution of 1024×1024. Diffusion models are a type of generative model based on Markov chain theory. They construct a forward diffusion process by progressively adding Gaussian noise to the data, and then learn a reverse denoising process to generate high-quality samples from random noise. The multi-view texture diffusion model uses ControlNet, an Adapter, or a conditional encoder to accept position map and normal map conditions. ControlNet represents the conditional input architecture used to control the output of the multi-view texture diffusion model.

[0037] The multi-view texture diffusion model starts with latent variables corresponding to each camera viewpoint. These latent variables are the data variables in the compressed latent space of the multi-view texture diffusion model. Texture images are generated progressively through several denoising time steps. In each denoising time step, generation conditional features are used to define the object content and material style that the texture should express at the current viewpoint. Among them, the position map provides the model with the 3D spatial position of the surface point corresponding to the current pixel, enabling a local texture to be stably attached to the corresponding surface region. The normal map provides the model with the local orientation of the surface corresponding to the current pixel, making the texture boundary, texture direction, and surface changes adapt to the orientation of the 3D curved surface. Camera intrinsic and extrinsic parameters enable the model to identify the viewing direction corresponding to different texture images. The position map and normal map together constitute dual geometric conditions, allowing the multi-view texture diffusion model to simultaneously obtain two types of information: texture generation position and local surface orientation.

[0038] To ensure consistent texture representation of the same 3D surface point across different camera viewpoints, consistency constraints are applied based on correspondences during the denoising timestep of the multi-view texture diffusion model. Consistency constraints can be executed at each denoising timestep or at intervals of several denoising timesteps. Execution at each timestep continuously constrains multi-view features. Feature aggregation involves finding the pixel positions of the same 3D surface point across all visible camera viewpoints from the correspondences, extracting the features of these pixels in the current denoising timestep, and then averaging or weighted averaging these features. Averaging is suitable for situations where visibility levels are similar across viewpoints, while weighted averaging can mitigate the influence of more reliable viewpoints based on visibility status and viewing angle.

[0039] Cross-attention uses features from a specific camera viewpoint as query features and corresponding features from other visible camera views as key and value features. It calculates the similarity between the query feature and each key feature, then weights and sums the value features according to their similarity. The weighted result is then used to update the features of the current camera viewpoint. Views with higher similarity have a greater impact on the current features, while views with lower similarity or partial occlusion have a smaller impact. By performing this process on each camera viewpoint separately, the multi-view texture diffusion model can exchange texture information of the same 3D surface point while maintaining viewpoint differences.

[0040] Shared noise is used to generate identical noise or correlated noise derived from the same base noise for corresponding positions of the same 3D surface points under different camera viewpoints. If different camera viewpoints generate random noise completely independently, even with the same generation conditions, significant differences in local textures may occur in the initial stages of denoising. By sharing noise, the same 3D surface points are denoised from a similar random state under each viewpoint, reducing the probability of generating irrelevant colors or details later. For surface points visible only from one camera viewpoint, cross-viewpoint sharing is not performed; instead, they are processed according to the normal noise of that camera viewpoint.

[0041] Consistency loss is used to constrain multi-view features during model training or backpropagation optimization. It extracts features of the same 3D surface point from different camera views based on correspondences, and pairs features from any two visible views. The differences between the feature components of each pair are calculated, squared, and summed to obtain the view difference of the 3D surface point. This summation of view differences across multiple 3D surface points forms the consistency loss. This consistency loss is then backpropagated together with the original diffusion loss of the multi-view texture diffusion model, causing the model parameters to be updated in a direction that reduces the cross-view feature differences of the same surface point. This reduces issues such as color inconsistency, boundary shifts, and changes in detail direction within the same texture region under different views.

[0042] In the S400, UV unwrapping is performed on the 3D geometry. UV unwrapping divides the 3D mesh surface into several regions that can be mapped to a 2D texture space, and assigns 2D UV coordinates to mesh vertices or surface points. This unfolds a continuous 3D surface into non-overlapping 2D planar regions, providing the geometric basis for texture mapping applications. The unfolded 2D regions are called UV islands, which are either non-overlapping or arranged according to pre-defined rules. UV unwrapping uses ABF++ or LSCM algorithms to minimize deformation of the 3D surface when unfolding it to the 2D plane. When projecting texture images from different camera views onto the same UV texture space, the corresponding facet number and centroid coordinates are first obtained from the surface point index map based on the pixel coordinates of the current texture image. Then, interpolation is performed using the vertex UV coordinates of the facet in the UV unwrapping result to obtain the UV position corresponding to the texture pixel. Thus, 2D pixels in the texture image are mapped to UV pixels via 3D surface points. Pixels with visibility masks are not included in the projection to avoid incorrect colors of occluded faces being written into the UV texture map. For cases where the projection location is between two UV pixels, bilinear interpolation or bicubic interpolation can be used to assign the texture color to the adjacent UV pixels.

[0043] When a UV pixel receives only a valid texture from one camera viewpoint, the texture color of that viewpoint is used as the candidate texture for that UV pixel. When textures from several camera views are projected onto the same UV pixel, the viewpoint angle weight, normal direction weight, visibility weight, and confidence score for each camera viewpoint are obtained. The closer the viewpoint angle of a camera viewpoint is to the vertical direction, the higher the viewpoint angle weight. The more perpendicular the camera viewpoint is to the viewing direction, the higher the normal direction weight. The visibility weight can be determined based on the visibility mask; the higher the visibility weight is for relatively fully visible areas. The confidence score is an uncertainty estimate of the multi-view texture diffusion model. All four types of weights are normalized to avoid using textures with obvious occlusion or uncertainty simply because one condition is high.

[0044] After denoising, each camera viewpoint yields an RGB texture image, where the RGB value of each pixel represents the texture color provided by that viewpoint. Once the fusion weights are determined, the texture color provided by each camera viewpoint is multiplied by its corresponding fusion weight. The weighted texture colors from all camera views are then summed and divided by the sum of the fusion weights for all camera views to obtain the fused texture color of the current UV pixel. This process is repeated for all pixels in the UV texture space to determine the initial UV texture map. If a particular UV pixel is invisible to all camera views, it is marked as an occluded area and not directly filled with color.

[0045] After the initial UV texture map is generated, the UV island boundaries, occlusion regions, and conflict regions are determined. UV island boundaries are the 2D boundaries formed when adjacent 3D surfaces are segmented into different UV islands during UV unwrapping. Occlusion regions are areas lacking texture due to being invisible from all camera views or insufficient effective projection. Conflict regions are areas where there are significant color or texture differences in the projection results from multiple camera views. Conflict regions can be determined based on the color and confidence differences of the textures from the participating views; no specific limitations are imposed here.

[0046] Texture dilation expands the effective texture within a UV island by 1 to 2 pixels towards the boundary or unfilled pixels to avoid black edges in texture sampling. Neighborhood completion samples color and details from adjacent pixels with continuous textures within the same UV island to complete the seam. For areas where corresponding textures can be found from other camera views, views with higher visibility and confidence are prioritized. Poisson blending handles areas where the colors on both sides of the seam are similar but there are slow changes in brightness or hue. It can be set to 100 to 500 iterations to ensure a smooth transition between the blended area and the textures on both sides. Diffusion repair handles large occluded areas or areas with missing content. It first expands the repair mask by 3 to 5 pixels outward, and then combines adjacent UV textures, corresponding textures from other camera views, and generation condition features for completion. These methods can be selected in one or more combinations depending on the region, and no specific limitations are made here. After the seam area correction is completed, the final target UV texture map is obtained. The target UV texture map is then bound to the 3D geometry according to the UV unwrapping results, allowing the 3D mesh surface to read the corresponding color and texture information from the target UV texture map during rendering. The final 3D asset includes 3D geometry, UV coordinates, and the target UV texture map. The target UV texture map is derived from the projection fusion of texture images from each camera's viewpoint and has undergone seam correction.

[0047] In one specific embodiment, the user inputs a text prompt to generate a mechanical part with a metallic texture. A text encoder extracts the object category, mechanical structure, and metallic material description to obtain text conditional features. The VAE-DiT architecture's 3D geometry generation model generates truncated symbolic distance-field volume data at a cubic resolution of 128, with the truncation distance set to 0.15 times the bounding box radius. A 3D mesh is then obtained through isosurface extraction. Twelve equally spaced camera views are set around the 3D mesh, and position maps, normal maps, depth maps, visibility masks, and surface point index maps are generated synchronously during the same rasterization process. Based on these, cross-view correspondences for the same 3D surface points are established.

[0048] Textual conditional features, position maps and normal maps from 12 viewpoints, and camera parameters are input into a multi-view texture diffusion model built on the SDXL architecture. Weighted feature aggregation is performed every 5 denoising time steps, and the accumulated sum of the squared differences of the features is used as the consistency loss, which is backpropagated along with the diffusion loss; the influence of the consistency loss is set to 0.5. After denoising, the 3D mesh is UV-unwrapped, and multi-view textures projected to the same UV pixels are fused using a four-class weighted product. One-pixel texture dilation is performed on UV island boundaries, and 200 Poisson fusions are performed on color discontinuities to obtain the target UV texture map, which is then bound to the 3D mesh.

[0049] In summary, by receiving text or image prompts and extracting generation condition features, and then setting several camera perspectives around the same 3D geometry, a unified and stable geometric foundation is provided for subsequent texture generation. By synchronously generating position maps and normal maps under the same camera parameters and pixel projection relationships, the two types of geometric conditions correspond to the same 3D surface point at the pixel level, avoiding alignment errors caused by differences in geometric sources, rendering steps, or projection parameters. Specifically, the position map defines the spatial attachment position of the texture on the 3D surface, while the normal map defines the local surface orientation and texture extension direction. Inputting these two as dual geometric conditions into the multi-view texture diffusion model improves the accuracy and directional consistency of texture positioning on complex surfaces, edges, and uneven areas. By applying consistency constraints to the features of the same 3D surface point under different camera perspectives, the colors, patterns, and material boundaries generated from different perspectives remain coordinated, reducing color abrupt changes, detail drift, and texture breaks that occur when switching perspectives. After projecting texture images from various camera perspectives into the same UV texture space, weighted fusion is performed based on the viewpoint angle, normal direction, visibility, and confidence level to reduce the impact of occlusion, oblique views, and low-confidence perspectives on the final texture map. UV island boundaries, occluded areas, and conflict areas are corrected through texture dilation, neighborhood completion, Poisson fusion, or diffusion repair, improving the continuity and integrity of the target UV texture map. This forms a complete processing flow from 3D geometry generation, dual geometry condition construction, multi-view texture diffusion, consistency constraints to UV fusion and seam correction, improving the automation level, texture stability, and visual quality of 3D asset generation.

[0050] Based on another preferred embodiment described above, see [link to preferred embodiment]. Figure 2 As shown, this embodiment provides a geometry-based 3D asset generation system for applying the above-described geometry-based 3D asset generation method, including: The geometry generation module is configured to receive and encode text or image prompts from the user, determine generation condition features, generate 3D geometry based on the generation condition features, and uniformly set several camera viewpoints around the 3D geometry, and perform synchronous rendering based on the same camera parameters. The synchronous generation module is configured to synchronously project the 3D geometry based on the camera parameters and pixel projection relationship of each camera viewpoint to determine the position map and normal map. The position map records the 3D spatial position of each pixel, and the normal map records the surface normal direction of each pixel. The condition construction module is configured to input the generated condition features, the position map and normal map of each camera view, and the camera parameters into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and normal map as dual geometric conditions for texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different views to determine the texture image of each camera view. The texture diffusion module is configured to perform UV unwrapping on the 3D geometry, project texture images from each camera viewpoint onto the same UV texture space, fuse textures projected onto the same UV pixels, determine the initial UV texture map, correct the seam areas of the initial UV texture map, and determine the 3D asset of the target UV texture map.

[0051] It is understandable that the above-mentioned geometrically based 3D asset generation method and system have the same beneficial effects, and will not be elaborated further here.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for generating 3D assets based on geometric conditions, characterized in that, include: Receive and encode text or image prompts input by the user, determine generation condition features, generate three-dimensional geometry based on the generation condition features, and uniformly set several camera viewpoints around the three-dimensional geometry, and perform synchronous rendering based on the same camera parameters. The three-dimensional geometry is synchronously projected based on the camera parameters and pixel projection relationship of each camera viewpoint to determine the position map and normal map. The position map records the three-dimensional spatial position of each pixel, and the normal map records the surface normal direction of each pixel. The generation condition features, the position map and normal map of each camera view, and the camera parameters are input into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and the normal map as dual geometric conditions to perform texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different views to determine the texture image of each camera view. The three-dimensional geometry is UV unwrapped, the texture images from each camera viewpoint are projected onto the same UV texture space, and the textures projected onto the same UV pixels are fused to determine the initial UV texture map. The seam areas of the initial UV texture map are corrected to determine the 3D asset of the target UV texture map.

2. The 3D asset generation method based on geometric conditions according to claim 1, characterized in that, When determining the generation condition features, the following are included: The text prompt is input into a pre-trained text encoder to extract the text semantics of the text prompt and encode them into text conditional features. The image prompt is input into a pre-trained image encoder to extract the image content, structure and style information of the image prompt and encode them into image conditional features. The generated conditional features include the text conditional features and the image conditional features.

3. The 3D asset generation method based on geometric conditions according to claim 2, characterized in that, When generating 3D geometry based on generation conditional features, the following are included: The 3D geometry generation model based on the VAE-DiT architecture denoises the generation condition features and generates truncated symbolic distance field volume data, and determines the three-dimensional geometry based on isosurface extraction. The resolution of the truncated symbol distance from the field volume is 128. 3 Or 256 3 The cutoff distance is 0.1-0.2 times the bounding box radius.

4. The 3D asset generation method based on geometric conditions according to claim 3, characterized in that, When determining the location map and normal map, the following should be included: Configure camera intrinsics and extrinsic parameters, and use the same camera intrinsics, camera extrinsic parameters, and rasterization rendering pipeline for each camera viewpoint to generate depth maps, visibility masks, surface point index maps, position maps, and normal maps; The location map and the normal map are derived from the same 3D geometry, camera viewpoint, and pixel projection relationship, and the same pixels in the location map and the normal map correspond to the same 3D surface point; The depth map is used to record the depth value of each pixel; The visibility mask is used to mark whether each pixel is visible; The surface point index map is used to record the patch number and centroid coordinates corresponding to each pixel.

5. The 3D asset generation method based on geometric conditions according to claim 4, characterized in that, When determining the texture images for each camera viewpoint, the following is included: The text conditional features, the position maps and normal maps of each camera viewpoint, the camera intrinsic parameters and the camera extrinsic parameters are input into the multi-view texture diffusion model, which is constructed based on the SDXL diffusion model architecture. The location map is used to constrain the three-dimensional spatial position of the texture, and the normal map is used to constrain the local orientation and texture direction. In each denoising time step of the multi-view texture diffusion model, the same three-dimensional surface points from different camera views are constrained to be consistent.

6. The 3D asset generation method based on geometric conditions according to claim 5, characterized in that, Determining the texture images for each camera viewpoint also includes: The consistency constraints include one or more combinations of constraint algorithms: The constraint algorithm includes feature aggregation, cross attention, shared noise, and consistency loss; The feature aggregation then averages or weighted averages the features of the same three-dimensional surface points from different camera perspectives. The cross-attention method uses features from one camera viewpoint as the Query and features from other camera viewpoints as the key. It then performs a dot product calculation based on the Query and the key to determine the similarity, and performs a weighted summation of the values ​​based on the similarity.

7. The 3D asset generation method based on geometric conditions according to claim 6, characterized in that, Determining the texture images for each camera viewpoint also includes: The shared noise generates the same noise for all camera views of the same three-dimensional surface point; The consistency loss extracts the features of 3D surface points under different camera views and forms feature pairs. It calculates the squared difference between each feature pair, accumulates all squared differences, and backpropagates the accumulated result together with the loss of the multi-view texture diffusion model.

8. The 3D asset generation method based on geometric conditions according to claim 7, characterized in that, When performing texture blending, the following are included: If several camera viewpoints correspond to the same UV pixels, then a fusion weight is calculated. The fusion weight is the product of the viewpoint angle weight, the normal direction weight, the visibility weight, and the confidence level. The confidence level is the uncertainty estimate of the multi-view texture diffusion model. The initial UV texture map is determined based on the fusion weight.

9. The 3D asset generation method based on geometric conditions according to claim 8, characterized in that, When correcting the seam areas of the initial UV texture map, the following steps are included: Determine the UV island boundaries, occlusion areas, and conflict areas in the initial UV texture map; Based on the correspondence of the same three-dimensional surface points, texture correction is performed on the UV island boundary, the occlusion area and the conflict area from the perspective of adjacent camera or adjacent UV region using one or more of the following methods: texture dilation, neighborhood completion, Poisson fusion and diffusion repair.

10. A geometry-based 3D asset generation system, used to apply the geometry-based 3D asset generation method as described in any one of claims 1-9, characterized in that, include: The geometry generation module is configured to receive and encode text or image prompts input by the user, determine generation condition features, generate three-dimensional geometry based on the generation condition features, and uniformly set several camera viewpoints around the three-dimensional geometry, and perform synchronous rendering based on the same camera parameters. The synchronous generation module is configured to synchronously project the three-dimensional geometry based on the camera parameters and pixel projection relationship of each camera viewpoint to determine the position map and normal map. The position map records the three-dimensional spatial position corresponding to each pixel, and the normal map records the surface normal direction corresponding to each pixel. The condition construction module is configured to input the generated condition features, the position map and normal map of each camera view, and camera parameters into the multi-view texture diffusion model. The multi-view texture diffusion model uses the position map and the normal map as dual geometric conditions to perform texture diffusion processing, and applies consistency constraints to the three-dimensional surface points in different views to determine the texture image of each camera view. The texture diffusion module is configured to perform UV unwrapping on the three-dimensional geometry, project texture images from each camera viewpoint onto the same UV texture space, fuse textures projected onto the same UV pixels, determine an initial UV texture map, correct the seam areas of the initial UV texture map, and determine the 3D asset of the target UV texture map.

Images