Three-dimensional asset generation method and system based on multi-view consistency constraint

Abstract

The present application belongs to the technical field of three-dimensional asset generation, and particularly relates to a three-dimensional asset generation method and system based on multi-view consistency constraint. The method comprises the following steps: obtaining a single reference image containing a target object; generating a multi-view image set corresponding to the target object by using a bidirectional recursive multi-view generation method; performing confidence filtering and spatial fusion on three-dimensional points obtained by projecting effective pixels in each view to obtain a coarse geometric proxy point cloud of the target object; performing voxelization processing on the coarse geometric proxy point cloud to obtain an initial sparse implicit structure of the target object; inputting the initial sparse implicit structure, the multi-view image and the corresponding camera pose into a multi-view guided optimization network to perform view-by-view iterative updating on the sparse implicit structure to obtain an optimized sparse implicit structure; and performing geometric decoding and appearance decoding on the optimized sparse implicit structure to generate a three-dimensional asset. The present application improves the three-dimensional structure recovery capability and surface appearance restoration capability of complex target objects.

Description

Technical Field

[0001] This invention belongs to the field of 3D asset generation technology, and particularly relates to multi-view... Figure 1 A method and system for generating 3D assets with consistency constraints. Background Technology

[0002] In AI-based single-image 3D asset generation technology, a single image only contains two-dimensional projection information of the target object in one viewing direction. The geometric structure and texture information of the target object's back, sides, and occluded areas cannot be directly observed. The same two-dimensional image can often correspond to multiple different 3D structures, making the single-image 3D generation process highly unconstrained and uncertain.

[0003] To alleviate the problem of insufficient information in a single image, existing technologies typically employ the following two types of technical solutions: The first type of technical solution first generates multiple two-dimensional images from different viewing directions based on a single reference image, and then uses neural radiation fields, symbolic distance fields, or other three-dimensional reconstruction methods to recover the three-dimensional model from the generated multi-view images; The second type of technical solution uses a feedforward three-dimensional generation network to directly predict three-dimensional representations such as three planes, sparse voxels, vector sets, three-dimensional Gaussian or implicit surfaces from a single reference image.

[0004] However: In the first type of scheme, there is a lack of reliable recursive dependencies and consistency constraints between different target views, which can easily lead to contradictory geometric structures and texture appearances. Furthermore, the unidirectional recursive generation process is prone to long-sequence error accumulation, making it difficult to simultaneously guarantee the continuity of adjacent views and long-distance views. Figure 1 In the second type of scheme, 3D assets are generated directly from a single image or random 3D representation. However, due to the lack of explicit coarse geometric priors, structural distortion, local missing parts, and geometric collapse are prone to occur. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention provides a multi-view based... Figure 1 The method and system for generating 3D assets with consistency constraints establish a continuous constraint chain from consistent multi-view generation, reliable coarse geometry construction, sparse implicit structure optimization to joint generation of geometry and texture. This reduces conflicting multi-view observations from the generation source and continuously transmits consistency requirements to the final 3D asset, which helps improve the overall stability of the generation results, enhances the ability to restore the 3D structure and surface appearance of complex target objects, and realizes the automated generation of high-quality textured 3D assets from a single reference image.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides a multi-view based Figure 1 A method for generating 3D assets with consistency constraints.

[0007] Based on multi-view Figure 1 The method for generating 3D assets with consistency constraints includes the following steps: Obtain a single reference image containing the target object; Based on the single reference image and the preset target camera pose, a set of multi-view images corresponding to the target object is generated using a bidirectional recursive multi-view generation method. Based on the multi-view image set and the camera pose corresponding to each view image, the depth information and confidence information of each view are estimated. The effective pixels in each view are projected onto a unified three-dimensional coordinate system, and the three-dimensional points obtained by projection are subjected to confidence filtering and spatial fusion to obtain the coarse geometric proxy point cloud of the target object. The coarse geometric proxy point cloud is voxelized to determine the spatial location of the effective voxels, and initial features are configured for each effective voxel to obtain the initial sparse implicit structure of the target object. The initial sparse implicit structure, multi-view images, and corresponding camera poses are input into the multi-view guided optimization network. Through the interaction between multi-view image features and the sparse implicit structure, the sparse implicit structure is iteratively updated view by view to obtain the optimized sparse implicit structure. The optimized sparse implicit structure is geometrically and visually decoded to generate 3D assets.

[0008] The second aspect of the present invention provides a multi-view based Figure 1 A 3D asset generation system with consistency constraints.

[0009] Based on multi-view Figure 1 A 3D asset generation system with consistency constraints includes: The reference image acquisition module is configured to acquire a single reference image containing the target object. The consistent multi-view generation module is configured to generate a set of multi-view images corresponding to the target object based on the single reference image and the preset target camera pose using a bidirectional recursive multi-view generation method. The coarse geometry proxy construction module is configured to: estimate the depth information and confidence information of each view based on the multi-view image set and the camera pose corresponding to each view image, project the effective pixels in each view to a unified three-dimensional coordinate system, and perform confidence filtering and spatial fusion on the projected three-dimensional points to obtain the coarse geometry proxy point cloud of the target object. The sparse implicit structure construction module is configured to: voxelize the coarse geometric proxy point cloud, determine the spatial location of the effective voxels, configure initial features for each effective voxel, and obtain the initial sparse implicit structure of the target object. The multi-view guided optimization module is configured to: input the initial sparse implicit structure, multi-view images and corresponding camera poses into the multi-view guided optimization network, and perform view-by-view iterative updates on the sparse implicit structure through the interaction between the multi-view image features and the sparse implicit structure to obtain the optimized sparse implicit structure. The 3D asset generation module is configured to perform geometric decoding and appearance decoding on the optimized sparse implicit structure to generate 3D assets.

[0010] The above one or more technical solutions have the following beneficial effects: This invention provides a multi-view-based Figure 1 A method and system for generating 3D assets with consistency constraints employs a bidirectional recursive multi-view generation approach. Starting from the initial viewing direction corresponding to a single reference image, it progressively generates target views along two opposite directions. The generation of the current view is constrained not only by the single reference image and the target camera pose but also by adjacent already generated views. This enhances the structural and texture continuity between adjacent views, reducing common problems in traditional parallel multi-view generation such as component variations, contour conflicts, color drift, and texture inconsistencies. Coarse geometric proxies are used to determine the spatial positions of sparse voxels, providing a clear spatial framework for subsequent 3D asset generation. Compared to generating directly from a random 3D representation or a complete 3D space, this invention can reduce the spatial scope requiring optimization, lower the difficulty of restoring complex 3D structures, and reduce problems such as component position offsets, scale distortion, local missing parts, and structural collapse.

[0011] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0012] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0013] Figure 1 This is a schematic diagram of the overall process of Embodiment 1 of the present invention.

[0014] Figure 2 This is a schematic diagram of the bidirectional recursive multi-view generation process according to Embodiment 1 of the present invention.

[0015] Figure 3 This is a schematic diagram of the system structure of Embodiment 2 of the present invention. Detailed Implementation

[0016] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0017] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0018] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0019] Example 1 With the development of fields such as virtual reality, augmented reality, digital twins, video games, and film and television production, the demand for high-quality 3D assets continues to grow. Traditional 3D assets rely on manual creation by professionals, which suffers from long cycles, high barriers to entry, and high costs. Artificial intelligence-based single-image 3D asset generation technology can automatically reconstruct the 3D geometry and surface appearance of a target object from a reference image, providing a new technological path to improve the efficiency of 3D content production.

[0020] However, a single image only contains two-dimensional information about the target object from a single viewing direction. The back, sides, and occluded areas of the target object lack direct observation, resulting in strong underconstraints and uncertainties in the 3D generation process. How to generate 3D geometry and surface textures that are consistent with the reference image and stable across multiple viewpoints under limited visual information conditions remains a core technical challenge in this field.

[0021] To alleviate the problem of insufficient information from a single image, existing techniques typically generate multiple target views based on a single reference image, and then use neural radiation fields, symbolic distance fields, or other 3D reconstruction methods to recover the 3D model. This approach can supplement information in invisible areas to some extent, but existing multi-view generation models often lack stable and continuous conditional dependencies between views, easily leading to conflicts in object outlines, local components, material colors, and texture details. If conflicting multi-view images are used for subsequent 3D reconstruction, problems such as 3D surface blurring, local structural collapse, missing components, texture ghosting, and abrupt changes in appearance during viewpoint switching can easily occur. Some methods use unidirectional autoregression or recursion to enhance the continuity of adjacent views, but error accumulation still exists when generating long sequences, making it difficult to guarantee global multi-view functionality. Figure 1 To the point of being responsive.

[0022] Another approach employs a feedforward 3D generation network, directly predicting the 3D representation from a single reference image. While this method is faster, it relies heavily on prior inference of invisible regions based on training data due to the lack of explicit multi-view observation constraints. When the target object has a complex structure, special components, or a unique appearance, the generated result is prone to discrepancies with the specific object represented by the reference image, leading to problems such as incorrect local structural proportions, missing small parts, unstable back-side structures, and texture / color shifts. Furthermore, some methods lack reliable coarse geometric priors or only fuse all view information at once, easily resulting in structural distortion, cross-view conflicts, and insufficient utilization of complementary information.

[0023] Furthermore, existing methods often separate geometry generation from texture appearance generation when generating 3D assets, lacking unified multi-view constraints. This can easily lead to mismatches between textures and geometric surfaces, resulting in problems such as texture misalignment, stretching, seams, and inconsistent multi-view rendering, making it difficult to meet the requirements of practical applications for high-quality textured 3D assets.

[0024] Therefore, in summary, existing single-image 3D asset generation technologies still have the following main technical shortcomings: First, the lack of reliable recursive dependencies and consistency constraints between different target views makes it easy to generate contradictory geometric structures and texture appearances. Secondly, the unidirectional recursive generation process is prone to long-sequence error accumulation, making it difficult to simultaneously guarantee the continuity of adjacent views and long-distance views. Figure 1 To the point of being compatible; Third, under the condition of sparse and error-prone multi-view observation, existing reconstruction methods are difficult to construct stable and reliable three-dimensional geometric initialization; Fourth, generating 3D assets directly from a single image or random 3D representation lacks explicit coarse geometric priors and is prone to structural distortion, local missing parts, and geometric collapse. Fifth, multi-view information is usually fused in parallel or all at once, making it difficult to gradually resolve conflicts between different views and make full use of complementary observation information; Sixth, the lack of a tightly coupled joint optimization mechanism between geometry and texture appearance can easily lead to problems such as texture mismatch and inconsistent rendering across multiple perspectives.

[0025] Based on this, this embodiment provides a multi-view-based Figure 1 The method and system for generating 3D assets with consistency constraints solves the technical problems in the existing single-image 3D asset generation process, such as contradictions between multiple view images, accumulation of recursive generation errors, instability of coarse geometry initialization, inaccurate restoration of invisible region structure, and inconsistency between geometry and texture. It improves the geometric integrity, texture fidelity and multi-view consistency of 3D assets under single-image conditions, and realizes the automated generation of high-quality textured 3D assets from a single reference image.

[0026] Specifically, this invention constructs a bidirectional recursive multi-view generation mechanism, which starts from a reference view and gradually expands the target view in two opposite directions. This allows the generation of the current view to be constrained by the reference image, adjacent generated views, and the pose of the target camera, thereby enhancing the structural continuity between adjacent views, shortening the recursive propagation path in a single direction, and reducing error accumulation in the long sequence generation process.

[0027] Based on this, the present invention obtains the depth and confidence information of each view using a feedforward geometric estimation method according to the generated multi-view images and their camera poses, and further obtains this information through low-confidence data filtering and multi-view... Figure 3 Dimension fusion constructs a coarse geometric proxy, providing a stable, reliable geometric initialization that matches multi-view observations for subsequent 3D generation, reducing geometric distortions caused by direct generation from random 3D structures.

[0028] Furthermore, this invention converts the coarse geometric proxy into a sparse implicit structure and adopts a view-by-view iterative optimization method to sequentially inject image features and camera pose information from different viewing directions into the sparse implicit structure, enabling the model to gradually integrate complementary geometric and appearance information provided by different views and alleviate feature conflicts between different views during continuous updates.

[0029] Finally, this invention obtains the three-dimensional surface structure and surface texture information of the target object through joint decoding of geometric representation and appearance representation, and maps the multi-view appearance information to a three-dimensional mesh, thereby generating a three-dimensional asset that maintains a high degree of consistency with the reference image in terms of geometric structure, texture appearance and multi-view rendering results.

[0030] Through the above technical solutions, this invention aims to establish a continuous constraint chain from consistent multi-view generation, reliable coarse geometry construction, sparse implicit structure optimization to joint generation of geometric textures, thereby improving the three-dimensional structure recovery capability and surface appearance restoration capability of complex target objects, and providing a high-quality three-dimensional asset generation foundation for applications such as virtual reality, augmented reality, game production, digital twins, e-commerce display and film and television content production.

[0031] To facilitate understanding, some of the terms used in this invention will be explained first.

[0032] A single reference image refers to a two-dimensional image containing the target object to be generated. A single reference image can be obtained by taking a picture with a camera, generating an image generation model, computer rendering, or by cropping from a video frame. Preferably, the target object is located in the main area of ​​the image, and the background is a solid color background, a transparent background, or a simplified background that has undergone foreground segmentation.

[0033] Multi-view images refer to a collection of two-dimensional images formed by observing the same target object from different virtual camera viewing directions. Each multi-view image corresponds to a target camera pose, which represents the camera's position and orientation relative to the target object.

[0034] Bidirectional recursive multi-view generation refers to recursively generating subsequent views along two opposite directions around the target object, starting from the initial viewing direction corresponding to a single reference image. In each recursive generation process, the current target view is constrained by the single reference image, adjacent generated views, and the current target camera pose.

[0035] Coarse geometric proxies are initial 3D geometric representations constructed based on depth estimation results from multi-view images and camera poses. Coarse geometric proxies do not require the complete recovery of all geometric details of the target object; their main purpose is to define the approximate spatial extent and overall structure of the target object, providing geometric initialization for subsequent high-quality 3D generation.

[0036] Sparse implicit structure refers to a 3D representation that stores features only in the effective spatial locations that the target object may occupy. Compared to a complete dense voxel representation, sparse implicit structure only retains effective voxels, which can reduce invalid computations in the 3D feature processing.

[0037] Three-dimensional assets include one or more of the following: three-dimensional mesh models, three-dimensional Gaussian models, implicit surface models, textured mesh models, or three-dimensional models with material properties.

[0038] Specifically, this embodiment is based on multi-view Figure 1 The method for generating 3D assets with consistency constraints includes the following steps: S1. Reference image acquisition steps: Acquire a single reference image containing the target object; S2. Consistent Multi-View Generation Steps: Based on a single reference image and multiple preset target camera poses, a bidirectional recursive multi-view generation method is used to generate a set of multi-view images corresponding to the target object. When generating any target view image, the single reference image, the generated view images adjacent to the current target view, and the camera pose corresponding to the current target view are used together as generation conditions to establish a recursive dependency relationship between adjacent views, so as to enhance the consistency of different views in terms of geometric structure and surface appearance. S3. Coarse geometric proxy construction steps: Based on the multi-view image set and the camera pose corresponding to each multi-view image, estimate the depth information and confidence information of each view, project the image pixels that meet the confidence conditions onto a unified three-dimensional coordinate system, and fuse the three-dimensional points corresponding to different views to obtain the coarse geometric proxy of the target object. S4. Sparse implicit structure construction steps: Voxelize the coarse geometric proxy, determine the effective voxel positions based on the distribution of the coarse geometric proxy in three-dimensional space, and configure initial features for each effective voxel to obtain an initial sparse implicit structure with coarse geometric prior. S5. Multi-view guided optimization steps: Input the multi-view images and their corresponding camera poses into the multi-view guided optimization network in a preset order. Interact with the current sparse implicit structure through the image features and camera pose features of each view. Perform iterative updates on the sparse implicit structure view by view, and gradually integrate complementary geometric information and appearance information under different viewing directions to obtain the optimized sparse implicit structure. S6. 3D asset generation steps: Perform geometric decoding and appearance decoding on the optimized sparse implicit structure to obtain the 3D surface geometry and appearance information of the target object, and generate textured 3D assets based on the 3D surface geometry and appearance information.

[0039] Furthermore, the bidirectional recursive multi-view generation method in step S2 specifically includes: Taking the initial observation direction corresponding to a single reference image as the starting direction, a first camera pose sequence distributed along a first rotation direction and a second camera pose sequence distributed along a second rotation direction are set around the target object, with the first rotation direction being opposite to the second rotation direction; Along the first rotation direction, using the previous view image as a local condition for the current view image, multiple first-direction view images are recursively generated; Along the second rotation direction, using the previous view image as a local condition for the current view image, multiple second-direction view images are recursively generated; The first-direction view image and the second-direction view image are combined to form a multi-view image set covering multiple viewing directions of the target object.

[0040] Furthermore, the process of generating the target view image in step S2 specifically includes: Visual features are extracted from a single reference image and adjacent generated view images, respectively. Encode the camera pose corresponding to the current target view to obtain camera pose features; Visual features and camera pose features are injected as conditional information into the denoising process of the diffusion generation network. Random noise is gradually denoised using a diffusion generation network to generate the current target view image.

[0041] In one alternative implementation, a single reference image and adjacent generated view images are feature extracted by a visual encoder. The camera pose is represented by quaternions, rotation matrices, translation vectors, or combinations thereof, and is converted into high-dimensional pose features through a feature mapping layer. The visual features and pose features are injected into a diffusion denoising network through a conditional attention mechanism.

[0042] Furthermore, the training process of the diffusion-generated network is supervised using a noise prediction loss. For the nth target view... Its generation process is represented as follows:

[0043] in, This represents the nth target view image. Indicates the nth target view Adjacent previous view image, Represents a single reference image. Let θ represent the camera pose corresponding to the nth target view, and let θ represent the model parameters of the diffusion generation network. Indicates the model parameters Given certain conditions, a probability distribution is generated.

[0044] The noise prediction loss is used to constrain the difference between the noise predicted by the diffusion generation network and the real noise applied to the current target view image, so that the generated current target view maintains both global consistency with a single reference image and local continuity with neighboring view images.

[0045] Furthermore, step S3 specifically includes the following sub-steps: S31. Depth and confidence estimation: Input the multi-view image set and the camera pose corresponding to each multi-view image into the feedforward geometric estimation network to obtain the depth map and pixel-wise confidence map corresponding to each view. S32, 3D back projection: Based on the depth map, camera intrinsic parameters and camera extrinsic parameters corresponding to each view, the 2D pixels in each view are back projected to a unified 3D coordinate system to obtain the 3D point cloud corresponding to each view; S33, Confidence Filtering: Set a confidence threshold based on the pixel-by-pixel confidence map, and remove pixels with confidence scores below the confidence threshold and their corresponding 3D points to reduce the impact of erroneous depth estimation on coarse geometric proxies; S34. Multi-view fusion: Align and fuse 3D point clouds corresponding to different views and filtered by confidence in a unified 3D coordinate system to obtain a coarse geometric proxy point cloud of the target object.

[0046] Furthermore, the feedforward geometric estimation network establishes pixel correspondences between different view images through cross-view feature interaction and jointly predicts the depth map and pixel-wise confidence map of each view. The feedforward geometric estimation network can utilize pre-trained general geometric priors to generate stable depth estimation results even with a limited number of views.

[0047] Furthermore, step S4 specifically includes: A three-dimensional voxel space is pre-established, and the three-dimensional points in the coarse geometry proxy are mapped to the corresponding voxel units; The spatial location of effective voxels is determined based on whether each voxel unit contains effective 3D points from the coarse geometry proxy. Initial features are configured for effective voxels. These initial features can be obtained by random sampling from a preset probability distribution, or by local geometric features, image features, or a combination of both from a coarse geometric proxy. The spatial location of effective voxels and their initial features are combined to form an initial sparse implicit structure.

[0048] By using coarse geometric proxies to determine the positions of effective voxels, the subsequent 3D generation process is carried out within a spatial range that matches the initial shape of the target object, avoiding the recovery of the target geometry from a random state in a complete 3D space.

[0049] Furthermore, the multi-view guided optimization network in step S5 includes multiple serially configured multi-view diffusion transformation modules, each of which includes an image feature extraction unit, a camera pose encoding unit, a cross-attention unit, a sparse space attention unit, and a feature transformation unit.

[0050] The image feature extraction unit is used to extract multi-scale visual features of the current view image; The camera pose encoding unit is used to map the camera pose corresponding to the current view into pose features; Cross-attention units are used to conditionally update voxel features in the current sparse implicit structure based on multi-scale visual features and pose features; Sparse spatial attention units are used to establish spatial feature interactions between effective voxels, aggregating local information and long-range structural information in sparse implicit structures. The feature transformation unit is used to normalize and nonlinearly map the updated voxel features.

[0051] Furthermore, in each multi-view diffusion transformation module, the images of each view and their corresponding camera poses are input one by one in a preset order; for each input view, the image features and pose features of that view are extracted first, and then the current sparse implicit structure is updated once using the image features and pose features; after all views have been traversed once, the updated sparse implicit structure is input into the next multi-view diffusion transformation module.

[0052] By serially superimposing multiple multi-view diffusion transformation modules, the geometric and appearance information provided by different views is injected into the sparse implicit structure in stages and layers, avoiding the direct mixing of all view information in a single update, which would lead to feature competition and cross-view conflicts.

[0053] Furthermore, the input order of multi-view images can be determined based on the spatial adjacency relationship of the camera pose, so that there is a continuous change in viewpoint between adjacent input views; alternative directions, circular traversal, or confidence-based sorting input methods can also be used to iteratively update the sparse implicit structure.

[0054] Furthermore, step S6 specifically includes: S61. Appearance Representation Decoding: Input the optimized sparse implicit structure into the Gaussian representation decoder to obtain the appearance representation of the target object composed of multiple three-dimensional Gaussian primitives. Each three-dimensional Gaussian primitive contains at least one or more parameters among spatial location, scale, orientation, transparency, and color features. S62. Geometric Representation Decoding: Input the optimized sparse implicit structure into the symbolic distance field decoder to obtain the symbolic distance field representation of the target object; S63. 3D Mesh Extraction: Based on the symbolic distance field representation, the 3D mesh of the target object is obtained by isosurface extraction. S64, Multi-view Appearance Rendering: Set multiple rendering camera poses around the target object, and generate multi-view rendered images or sequences of rendered images of the target object based on the three-dimensional Gaussian representation; S65. Texture Mapping: Based on the projection relationship between multi-view rendered images, rendering camera poses, and 3D meshes, the appearance information in multi-view rendered images is mapped to 3D meshes to generate textured 3D assets.

[0055] Furthermore, isosurface extraction methods include, but are not limited to, the moving cube algorithm; texture mapping includes, but is not limited to, texture projection based on view visibility, multi-view texture fusion based on confidence weighting, texture atlas generation, and texture seam optimization.

[0056] Furthermore, when training the bidirectional recursive multi-view generation module and the multi-view guided optimization network, a joint loss function is used for supervision. The joint loss function includes at least two of the following: color reconstruction loss, structural similarity loss, perception loss, noise prediction loss, and geometric surface loss.

[0057] Color reconstruction loss is used to constrain the pixel color differences between the generated or rendered image and the real image; Structural similarity loss is used to constrain the local structural differences between generated or rendered images and real images; Perceptual loss is used to constrain the difference between generated or rendered images and real images in the deep visual feature space; Noise prediction loss is used to supervise the diffusion denoising process; Geometric surface loss is used to constrain the difference between the symbolic distance field or 3D surface obtained from decoding and the geometric surface of the real target object.

[0058] In a preferred embodiment, the joint loss function is expressed as:

[0059] in, Indicates color reconstruction loss. Represents structural similarity loss. Indicates perceived loss. Indicates the noise prediction loss. This represents the geometric surface loss; each loss term can be set with the same or different weight coefficients depending on the training task.

[0060] Furthermore, such as Figure 1 As shown, this embodiment provides a multi-view based Figure 1 The method for generating 3D assets with consistency constraints is described in more detail below.

[0061] S1. Reference image acquisition.

[0062] Obtain a single reference image I containing the target object to be generated.

[0063] In this embodiment, a single reference image is an RGB image and may include animal figures, human figures, furniture, daily necessities, architectural models, toys, or other target objects with independent three-dimensional structures. The single reference image may be pre-processed with target detection, foreground segmentation, background removal, center cropping, and size normalization to ensure the target object is located in the center region of the image.

[0064] Specifically, the foreground mask of the target object can be extracted first using a foreground segmentation model. The background region is then removed based on the foreground mask. Next, the image is cropped according to the bounding rectangle of the target object, while retaining a boundary region of a preset ratio around the target object. After that, the cropped image is scaled to a preset input resolution, and the pixel values ​​are normalized to a preset numerical range to obtain a standardized single reference image.

[0065] S2, Consistent Multi-View Generation.

[0066] Based on a single reference image I and a preset target camera pose, a set of multi-view images X of the target object is generated using a bidirectional recursive multi-view generation method.

[0067] like Figure 2 As shown, in this embodiment, the frontal viewing direction corresponding to a single reference image is taken as the starting direction, and two sets of target camera poses are set around the target object along the first rotation direction and the second rotation direction, wherein the first rotation direction is opposite to the second rotation direction.

[0068] In one specific embodiment, a total of 16 target views are set, with 8 target views set in the first rotation direction and 8 target views set in the second rotation direction. Each target camera is distributed around the center of the target object along an approximately circular trajectory, and there is a preset angular interval between adjacent target cameras.

[0069] In this embodiment, as Figure 2 As shown, the first rotation direction generates a leftward view. Figure 1 Left view Figure 2 Left view Figure 3 ...leftward view N, resulting in the first view sequence; second rotation direction generates the rightward view. Figure 1 Looking to the right Figure 2 Looking to the right Figure 3 ...From the rightward view N, a second view sequence is obtained, where N represents the number of views in either the first or second view sequence; in this embodiment, N is set to 8. The first and second view sequences are then combined into a consistent multi-view image set.

[0070] It is understandable. Figure 2 The conditional constraints refer to the constraint information used in the target view generation process, including a single reference image, adjacent generated view images, and the camera pose corresponding to the current target view. The single reference image provides an overall structural and appearance reference for the target object; adjacent generated view images constrain the structural and textural continuity between adjacent viewpoints; and the camera pose corresponding to the current target view limits the viewing direction of the target view. These conditional constraints ensure that the generated multiple target views maintain consistency in geometry and surface appearance.

[0071] The above number of views and angle intervals are only a preferred configuration, and can also be set to 8, 12, 24 or other numbers depending on the complexity of the target object.

[0072] In the first rotation direction, a first target view is generated using a single reference image as the initial view; subsequently, a second target view is generated using the first target view as the adjacent generated view, and subsequent target views are generated recursively in the same manner.

[0073] In the second rotation direction, a single reference image is used as the initial view, and multiple target views are recursively generated in the opposite direction to the first rotation direction.

[0074] For the nth target view image Its generation process can be represented as: , in, Indicates the nth target view Adjacent generated view images, where I represents a single reference image. This represents the camera pose corresponding to the nth target view. This represents the model parameters of the multi-view generation network. Indicates the model parameters Given certain conditions, a probability distribution is generated.

[0075] Specifically, a visual encoder is first used to extract individual reference images. and adjacent generated view images Visual features. Visual encoders can employ contrastive language image pre-trained models, visual Transformer networks, convolutional neural networks, or other image feature extraction networks.

[0076] Then, the target camera pose is determined. The camera pose is converted into a pose vector. The camera pose can be represented by quaternions, rotation matrices, Euler angles, camera center coordinates, or a combination of the above parameters, and is converted into a high-dimensional pose feature that matches the visual feature dimension through a linear mapping layer.

[0077] Features from a single reference image, features from adjacent generated views, and the target camera pose features are input into a conditional diffusion generation network. The network uses random Gaussian noise as initial latent variables, injects these conditional features into a denoising network through a conditional attention mechanism, and gradually removes noise through a multi-step backdiffusion process to generate the current target view image.

[0078] A single reference image provides global constraints on the target object's category, shape, and appearance; adjacent generated views provide local view continuity constraints; and the target camera pose controls the viewing direction of the currently generated image. These three elements work together to ensure that the current target view maintains overall consistency with the reference image and is continuous with adjacent views in its local structure.

[0079] By employing two recursive generation sequences in opposite directions, the complete field of view can be divided into two shorter recursive propagation paths, reducing error accumulation in a single long sequence. Once the views in both directions have been generated, they are combined according to the camera pose order to obtain a multi-view image set.

[0080] S3, coarse geometry proxy construction.

[0081] The multi-view image set X generated in step S2 and its corresponding camera pose set are input into the feedforward geometric estimation network to estimate the depth map and pixel-wise confidence map corresponding to each view.

[0082] Specifically, for the i-th view image The feedforward geometric estimation network outputs a depth map. and confidence plot Depth map A confidence map is used to represent the spatial depth corresponding to each pixel in an image. Used to indicate the reliability of the corresponding depth estimate.

[0083] The feedforward geometric estimation network establishes pixel correspondences between different views through cross-view feature interaction, and uses the 3D geometric priors obtained through pre-training to jointly predict the depth information of each view under the condition of limited views.

[0084] For the i-th view image Pixels u indicates that the pixel is in the th position. i The horizontal coordinate of a pixel in the i-th view image, v represents the vertical coordinate of that pixel in the i-th view image, based on its depth value. Camera intrinsic parameter matrix and camera extrinsic matrix The pixels are back-projected onto a unified three-dimensional world coordinate system. The three-dimensional point obtained by backprojection can be represented as: .

[0085] in, The pixel coordinates in the i-th view are represented by... Three-dimensional points obtained by back projection; This represents the camera intrinsic parameter matrix corresponding to the i-th view; This represents the camera extrinsic parameter matrix corresponding to the i-th view; Represents the camera intrinsic parameter matrix The inverse matrix; Represents the camera extrinsic parameter matrix The inverse matrix; Represents pixel coordinates a homogeneous column vector of coordinates; Represents pixel coordinates The corresponding depth value.

[0086] Before performing back projection, based on the confidence plot Set a confidence threshold τ. When the confidence level of a pixel is lower than the confidence threshold, the depth estimate corresponding to that pixel is considered unreliable, and the pixel and its corresponding 3D point are discarded.

[0087] In one specific implementation, a normalized confidence threshold of 0.95 is used to filter low-confidence 3D points. The threshold can also be set to other values ​​depending on the output range of different feedforward geometric estimation networks.

[0088] The 3D points filtered by confidence level from each view are projected onto a unified coordinate system, and point cloud fusion is performed directly. Further, radius filtering, statistical outlier removal, voxel downsampling, or multi-view techniques can be employed. Figure 1 The consistency check method removes isolated points and noise points that are significantly deviated from the main surface, and finally obtains the coarse geometric proxy point cloud P of the target object.

[0089] Coarse geometric proxy point clouds primarily reflect the overall outline, main components, and spatial extent of the target object. Even if its local surfaces are not yet complete, they can provide a stable initial geometric framework for subsequent sparse 3D structure optimization.

[0090] S4, Sparse implicit structure construction.

[0091] Voxelization is performed on the coarse geometric proxy point cloud P to construct the initial sparse implicit structure of the target object.

[0092] Specifically, a 3D bounding box is first established based on the spatial boundary of the coarse geometric proxy point cloud, and then the 3D bounding box is divided into a voxel mesh of a preset resolution. In one specific implementation, the voxel mesh resolution is 64×64×64.

[0093] Each 3D point in the coarse geometric proxy point cloud is mapped to a corresponding voxel, and the position of the effective voxel is determined based on whether each voxel contains a valid 3D point. For a voxel containing one or more valid 3D points, it is marked as an occupied voxel; for voxels located within a preset neighborhood of the occupied voxel, they can also be marked as candidate effective voxels as needed, thus reserving a certain range of space for adjustment in subsequent geometric refinement.

[0094] For each valid voxel, an initial feature vector of a preset dimension is configured. In this embodiment, the initial features are randomly sampled from a Gaussian distribution. Since the position of the valid voxel is determined by a coarse geometric proxy, the entire sparse implicit structure still has a coarse geometric prior to the target object, even if the voxel features are randomly initialized.

[0095] In other implementations, the initial features can also be initialized based on point cloud coordinates, normal vectors, color information, multi-view image features, or combinations thereof within the voxel neighborhood.

[0096] The spatial coordinates of all effective voxels and their corresponding features are combined to form an initial sparse implicit structure. .

[0097] S5, multi-view guided iterative optimization.

[0098] The initial sparse implicit structure obtained in step S4 The multi-view guided optimization network, obtained from the multi-view images and camera pose inputs in step S2, performs view-by-view iterative updates on the sparse implicit structure.

[0099] In this embodiment, the multi-view guided optimization network consists of multiple serially configured multi-view diffusion transformation modules. Each multi-view diffusion transformation module includes an image feature extraction unit, a camera pose encoding unit, a cross-attention unit, a sparse spatial attention unit, and a feedforward feature transformation unit.

[0100] In one specific implementation, the multi-view guided optimization network includes 24 serially configured multi-view diffusion transformation modules.

[0101] For any multi-view diffusion transform module, the first view image and its corresponding camera pose are first input. The visual features of the first view image are extracted using an image feature extraction network, and the first view camera pose is encoded into pose features using a linear mapping layer.

[0102] Subsequently, a cross-attention mechanism is used to establish the association between the visual features of the first view and the current sparse implicit structure. Sparse voxel features are used as query features, while the visual features and pose features of the current view are used as conditional features, enabling sparse voxels to extract geometric and appearance information related to their spatial position from the current view.

[0103] After cross-attention updates, sparse spatial attention is used to propagate structural information among effective voxels. Sparse spatial attention is computed only on effective voxels or their neighborhoods, and is used to model the local geometric relationships between adjacent voxels and the overall structural connections between voxels that are far apart.

[0104] After completing the feature injection for the first view, the updated sparse implicit structure is used as the current structure. Then, the second view image and its corresponding camera pose are input, and the structure is updated again in the same way.

[0105] After iterating through all the multi-view images, the output structure of the current multi-view diffusion transformation module is obtained. Then, this output structure is input into the next multi-view diffusion transformation module, and all views are iterated through again.

[0106] By serially stacking multiple modules, geometric and appearance information from different viewing directions is gradually fused into a sparse implicit structure. Compared to inputting all multi-view images into a single attention layer in parallel at once, this implementation reduces the competition between different view conditions in a single update and gradually resolves local conflicts between views.

[0107] After passing through all the multi-view diffusion transformation modules, the optimized sparse implicit structure Z is obtained.

[0108] S6, 3D asset generation.

[0109] The optimized sparse implicit structure Z is subjected to geometric decoding and appearance decoding respectively to obtain textured 3D assets.

[0110] First, the optimized sparse implicit structure Z is input into a 3D Gaussian representation decoder to obtain a representation of the target object's appearance composed of multiple 3D Gaussian primitives. Each 3D Gaussian primitive can include parameters such as spatial position, scale, rotation direction, opacity, and color features.

[0111] Based on 3D Gaussian representation, multiple virtual rendering cameras are set up around the target object, and 3D Gaussian primitives are projected and blended to generate a multi-view rendering image sequence of the target object.

[0112] In one specific implementation, 100 frames of images are rendered around the target object to form a dense sequence of viewpoints covering the main surface of the target object.

[0113] Simultaneously, the optimized sparse implicit structure Z is input into the symbolic distance field decoder to generate the symbolic distance field corresponding to the target object. Each spatial location in the symbolic distance field has a distance value, which represents the signed distance from that location to the target surface.

[0114] In one specific implementation, the output resolution of the symbolic distance field is 512×512×512. Based on the position of the zero isosurface in the symbolic distance field, a moving cube algorithm is used to extract a high-resolution 3D mesh of the target object.

[0115] After obtaining the 3D mesh and multi-view rendered image sequence, the projection relationship between the rendered image pixels and the 3D mesh surface is established based on the camera pose of each rendered image. For cases where the same mesh surface is observed by multiple views, the color information provided by different views can be weighted and fused according to the viewing angle, occlusion relationship, rendering transparency, image sharpness, or preset confidence level.

[0116] Furthermore, UV unwrapping can be performed on the 3D mesh to generate a 2D texture atlas, and color information from multi-view images can be written into the texture atlas. For texture seams between different views, color correction, Poisson blending, boundary smoothing, or learning-based texture optimization methods can be used for processing.

[0117] Finally, the generated texture atlas and material parameters are bound to the 3D mesh to obtain a 3D asset with complete geometry and consistent surface texture.

[0118] 3D assets can be saved as OBJ, FBX, GLB, GLTF, PLY, or other 3D file formats, or as 3D Gaussian representation, symbolic distance field representation, or other implicit 3D representations. Specifically, OBJ represents Wavefront OBJ 3D model file format, FBX represents Filmbox 3D asset exchange file format, GLB represents the binary form of glTF 3D asset file format, GLTF represents GL Transmission Format 3D asset transmission format, and PLY represents Polygon FileFormat or Stanford Triangle Format 3D data file format.

[0119] Network training process: The bidirectional recursive multi-view generation network and the multi-view guided optimization network in this embodiment can be trained using a dataset containing multi-view rendered images, camera poses, and real 3D models.

[0120] During training sample construction, for each real 3D object, multiple camera poses are set around it, and RGB images, depth maps, and camera parameters are rendered under different viewing directions. At the same time, the real mesh, symbolic distance field, or occupancy field of the 3D object are retained as geometric supervision information.

[0121] For a bidirectional recursive multi-view generation network, a view is selected from the multi-view images of each object as a reference image, and two adjacent view sequences are constructed in clockwise and counterclockwise directions. During training, the current target view and its previous view are randomly selected. The reference image, the previous view, and the target camera pose are used as conditions to add random noise to the current target view, and the network is trained to predict the added noise.

[0122] Noise prediction loss It can be represented as: , in, This represents the mathematical expectation operation, where ε represents the amount of real noise added to the current target view. This represents the noise predicted by the diffusion generation network. Indicates at the diffusion moment Noisy images, This represents the generated view image adjacent to the nth target view. Represents a single reference image. This represents the time step in the diffusion process. This represents the camera pose corresponding to the nth target view. Represents the L2 norm operation. This represents the L2 norm square operation, used to measure the error between the predicted noise and the actual noise in a diffusion generator network.

[0123] For the multi-view guided optimization network, the coarse geometric proxy voxels constructed based on the multi-view images are converted into an initial sparse implicit structure, and multiple views and their camera poses are sequentially input into the multi-view guided optimization network.

[0124] The optimized sparse implicit structure is processed by a 3D Gaussian representation decoder and a symbolic distance field decoder to obtain the rendered image and the 3D surface. The rendered image is compared with the real view image, and the generated symbolic distance field or 3D surface is compared with the geometry of the real object.

[0125] In one specific implementation, the following joint loss function is used. : , in, Indicates color reconstruction loss. Represents structural similarity loss. Indicates perceived loss. Indicates the noise prediction loss. This represents the loss of geometric surface.

[0126] Specifically: Color reconstruction loss is used to measure the pixel color error between a generated or rendered image and a real image. Structural similarity loss is used to constrain the local structural consistency between generated or rendered images and real images; Perceptual loss is used to constrain the consistency between generated or rendered images and real images in the deep visual feature space. Noise prediction loss is used to measure the prediction error between the noise predicted by the diffusion generation network and the actual noise. Geometric surface loss is used to constrain the surface consistency between the generated symbolic distance field or the geometry of the 3D surface and the real object.

[0127] Each loss term can be assigned the same or different weight coefficients depending on the training task.

[0128] In one specific implementation, the network is trained using the Adam optimizer, with an initial learning rate set to... The total batch size is set to 4. The bidirectional recursive multi-view generation module generates 16 multi-view images, the feedforward geometric estimation module keeps the parameters frozen after training, and the trainable parameters in the multi-view guided optimization network are updated according to the joint loss function.

[0129] The specific number of views, voxel resolution, number of network layers, number of rendered images, and training parameters mentioned above are all preferred configurations in this embodiment and do not constitute a limitation on the scope of protection of this invention.

[0130] Compared with the prior art, the present invention has the following significant advantages: 1. This invention establishes a consistency constraint chain that runs through the entire process of multi-view image generation, coarse geometry proxy construction, sparse 3D structure optimization, and geometric texture decoding. Compared with existing schemes that only apply constraints in the 3D reconstruction stage or the final rendering stage, this invention can reduce conflicting multi-view observations from the generation source and continuously pass on consistency requirements to the final 3D asset, which is beneficial to improving the overall stability of the generation results.

[0131] 2. This invention employs a bidirectional recursive multi-view generation method, starting from the initial viewing direction corresponding to a single reference image and progressively generating the target view along two opposite directions. The generation of the current view is constrained not only by the single reference image and the target camera pose, but also by adjacent already generated views. This enhances the structural and texture continuity between adjacent views and reduces common problems in traditional parallel multi-view generation, such as component variations, contour conflicts, color drift, and texture inconsistencies.

[0132] 3. Compared with the unidirectional recursive generation method, this invention distributes the recursive generation task within the complete view range in two opposite directions, shortening the propagation distance of a single recursive sequence. This reduces the continuous accumulation of errors from preceding views in long sequences, alleviates structural distortion and appearance shifts in distant views, and thus balances the continuity of adjacent views and global multi-view functionality. Figure 1 To the point of being responsive.

[0133] 4. This invention employs a feedforward geometric estimation method, jointly estimating depth information and pixel-by-pixel confidence information based on multi-view images and camera pose, and removing low-confidence geometric data before 3D fusion. This method can reduce the impact of erroneous depth, generated artifacts, and unreliable regions on 3D geometric initialization, constructing a relatively stable coarse geometric proxy under sparse multi-view conditions, and avoiding overfitting, surface floating objects, and local geometric collapse caused by insufficient observation in traditional iterative reconstruction methods.

[0134] 5. This invention utilizes coarse geometric proxies to determine the spatial positions of sparse voxels, providing a clear spatial framework for subsequent 3D asset generation. Compared to generating directly from a random 3D representation or a complete 3D space, this invention can narrow down the spatial range requiring optimization, reduce the difficulty of restoring complex 3D structures, and minimize problems such as component position offsets, scale distortion, local missing parts, and structural collapse.

[0135] 6. This invention employs a view-by-view iterative optimization method, sequentially injecting image features and camera pose features from different viewing directions into a sparse implicit structure. Compared to schemes that fuse all multi-view information in parallel during a single update, this method can absorb complementary information from each view in stages and gradually handle geometric and appearance conflicts between different views during continuous updates, which is beneficial for restoring invisible areas and complex local structures.

[0136] 7. This invention propagates local and long-distance structural information between effective voxels through sparse spatial attention, which can maintain the computational advantages of sparse 3D representation and enhance the spatial connection between different parts of the target object, which is beneficial for recovering slender structures, small parts, recessed areas and complex topological structures.

[0137] 8. This invention performs Gaussian appearance decoding and symbolic distance field geometric decoding on the same optimized sparse implicit structure, enabling the geometric structure and surface appearance to share unified multi-view conditions and three-dimensional implicit representation. This enhances the correspondence between texture and geometric surface, reducing texture misalignment, stretching, seams, and abrupt appearance changes during viewpoint switching caused by the separation of geometry generation and texture generation.

[0138] 9. This invention utilizes a symbolic distance field to extract a high-resolution 3D mesh, and uses 3D Gaussian representation to generate dense multi-view appearance observations. The appearance observations are then mapped to the 3D mesh, which can balance the integrity of the geometric surface and the detailed representation of the surface texture, generating textured 3D assets that can be directly used for rendering, editing, and content creation.

[0139] 10. This invention only requires a single image containing the target object as initial input to automatically complete multi-view image generation, 3D geometry estimation, implicit structure optimization, mesh extraction and texture mapping. It reduces the dependence of traditional 3D modeling on professional modelers and complex acquisition equipment, improves the efficiency of 3D asset production, and is applicable to application scenarios such as virtual reality, augmented reality, video games, film and television production, digital twins, e-commerce display and digital cultural content production.

[0140] 11. The number of target views, camera trajectory, coarse geometric representation, sparse implicit structure resolution, geometric decoding method and texture mapping method in this invention can all be adjusted according to different application scenarios, and have strong scalability and adaptability. They can be used for the generation of 3D assets of target objects of different categories, different complexities and different surface materials.

[0141] Example 2 This embodiment discloses a multi-view based method. Figure 1 A 3D asset generation system with consistency constraints.

[0142] like Figure 3 As shown, based on multi-view Figure 1 A 3D asset generation system with consistency constraints includes: The reference image acquisition module is configured to acquire a single reference image containing the target object. The consistent multi-view generation module is configured to generate a set of multi-view images corresponding to the target object based on a single reference image and a preset target camera pose, using a bidirectional recursive multi-view generation method. The coarse geometry proxy construction module is configured to: estimate the depth information and confidence information of each view based on the multi-view image set and the camera pose corresponding to each view image, project the effective pixels in each view to a unified three-dimensional coordinate system, and perform confidence filtering and spatial fusion on the projected three-dimensional points to obtain the coarse geometry proxy point cloud of the target object. The sparse implicit structure construction module is configured to: voxelize the coarse geometric proxy point cloud, determine the spatial location of the effective voxels, configure initial features for each effective voxel, and obtain the initial sparse implicit structure of the target object. The multi-view guided optimization module is configured to: input the initial sparse implicit structure, multi-view images and corresponding camera poses into the multi-view guided optimization network, and perform view-by-view iterative updates on the sparse implicit structure through the interaction between the multi-view image features and the sparse implicit structure to obtain the optimized sparse implicit structure. The 3D asset generation module is configured to perform geometric decoding and appearance decoding on the optimized sparse implicit structure to generate 3D assets.

[0143] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0144] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating 3D assets based on multi-view consistency constraints, characterized in that, Includes the following steps: Obtain a single reference image containing the target object; Based on the single reference image and the preset target camera pose, a set of multi-view images corresponding to the target object is generated using a bidirectional recursive multi-view generation method. Based on the multi-view image set and the camera pose corresponding to each view image, the depth information and confidence information of each view are estimated. The effective pixels in each view are projected onto a unified three-dimensional coordinate system, and the three-dimensional points obtained by projection are subjected to confidence filtering and spatial fusion to obtain the coarse geometric proxy point cloud of the target object. The coarse geometric proxy point cloud is voxelized to determine the spatial location of the effective voxels, and initial features are configured for each effective voxel to obtain the initial sparse implicit structure of the target object. The initial sparse implicit structure, multi-view images, and corresponding camera poses are input into the multi-view guided optimization network. Through the interaction between multi-view image features and the sparse implicit structure, the sparse implicit structure is iteratively updated view by view to obtain the optimized sparse implicit structure. The optimized sparse implicit structure is geometrically and visually decoded to generate 3D assets.

2. The method for generating 3D assets based on multi-view consistency constraints as described in claim 1, characterized in that, A bidirectional recursive multi-view generation method is used to generate a set of multi-view images corresponding to the target object, specifically including: Taking the initial observation direction corresponding to the single reference image as the starting direction, multiple target camera poses are set along the first rotation direction of the target object's circumference and the second rotation direction opposite to the first rotation direction, respectively. In the first rotation direction, the next target view image is recursively generated based on the previous target view image to form a first view sequence; In the second rotation direction, a subsequent target view image is recursively generated based on the previous target view image to form a second view sequence; The first view sequence is combined with the second view sequence to obtain a multi-view image set covering multiple viewing directions of the target object; or, Multiple target camera poses set in the first and second rotation directions are distributed around the center of the target object along a circular trajectory. or, The next target view image is generated recursively based on the previous target view image. n Target view image The generation process is represented as: ， in, Indicates the relationship with the first n The generated view images adjacent to the target view image. Represents a single reference image. Indicates the first n The camera pose corresponding to each target view. This represents the model parameters of the multi-view generation network. Indicates the model parameters Given certain conditions, a probability distribution is generated.

3. The method for generating 3D assets based on multi-view consistency constraints as described in claim 2, characterized in that, The process of generating any target view image to be generated includes: Visual features are extracted from the single reference image and the generated view image adjacent to the target view to be generated, respectively. The camera pose corresponding to the target view to be generated is encoded to obtain the camera pose features; The visual features of a single reference image, the visual features of the generated view image adjacent to the target view to be generated, and the camera pose features are used as conditional features input into the diffusion generation network. The conditional features are then injected into the diffusion denoising process through a conditional attention mechanism. The random noise is gradually denoised to obtain the target view image to be generated; or, The training process of the diffusion generation network is supervised by a noise prediction loss, which is used to constrain the difference between the predicted noise and the real noise applied to the current target view image.

4. The method for generating 3D assets based on multi-view consistency constraints as described in claim 1, characterized in that, Estimate the depth and confidence information for each view, specifically including: The multi-view image set and the camera pose corresponding to each view image are input into the feedforward geometric estimation network to predict the depth map and pixel-by-pixel confidence map of each view. The feedforward geometric estimation network establishes pixel correspondences between different view images through cross-view feature interaction, and jointly estimates depth information and confidence information based on the pixel correspondences. or, The process of obtaining the coarse geometric proxy point cloud is as follows: Based on the depth map and camera pose corresponding to each view, the two-dimensional pixels in each view are back-projected to a unified three-dimensional coordinate system to obtain point clouds of multiple views. A confidence threshold is set based on the pixel-by-pixel confidence map, and three-dimensional points with a confidence level lower than the confidence threshold are removed; Spatially align and merge the 3D points preserved in different views to obtain a coarse geometric proxy point cloud of the target object.

5. The method for generating 3D assets based on multi-view consistency constraints as described in claim 1, characterized in that, The process of obtaining the initial sparse implicit structure of the target object is as follows: Discretize the coarse geometric proxy point cloud to a 3D voxel space of a preset resolution, and map each 3D point in the coarse geometric proxy point cloud to the corresponding voxel. The spatial location of effective voxels is determined based on whether each voxel contains effective three-dimensional points. For each valid voxel, configure an initial feature vector of a preset dimension; Combine the spatial locations of all effective voxels and their corresponding initial features to form an initial sparse implicit structure; or, The initial features of the effective voxels are obtained by random sampling from a preset probability distribution.

6. The method for generating 3D assets based on multi-view consistency constraints as described in claim 3, characterized in that, The multi-view guided optimization network comprises multiple serially configured multi-view diffusion transformation modules; each multi-view diffusion transformation module includes an image feature extraction unit, a camera pose encoding unit, a cross-attention unit, a sparse space attention unit, and a feedforward feature transformation unit, wherein: The image feature extraction unit is used to extract the visual features of the current view image; The camera pose encoding unit is used to map the camera pose corresponding to the current view into pose features; The cross-attention unit is used to update the current sparse implicit structure based on the visual features and pose features; The sparse spatial attention unit is used to propagate and aggregate spatial structure information among sparse voxels; or, In each of the multi-view diffusion transformation modules, the images of each view and the corresponding camera poses are input sequentially according to a preset order, and the current sparse implicit structure is updated once after each view is input. After all views have been traversed, the updated sparse implicit structure is input into the next multi-view diffusion transformation module to achieve multi-level view-by-view iterative optimization.

7. The method for generating 3D assets based on multi-view consistency constraints as described in claim 6, characterized in that, The data processing procedure for multi-view guided optimization networks is as follows: For any multi-view diffusion transform module, first input the first view image and the corresponding camera pose, extract the visual features of the first view image, and encode the first view camera pose into pose features; Subsequently, a cross-attention mechanism is used to establish the association between the first-view visual features and the current sparse implicit structure; After cross-attention update, sparse spatial attention is used to propagate structural information among effective voxels; After completing the feature injection of the first view, the updated sparse implicit structure is used as the current structure. Then, the second view image and the corresponding camera pose are input, and the structure is updated again in the same way. After traversing all the multi-view images in sequence, the output structure of the current multi-view diffusion transformation module is obtained; Then, the output structure of the current multi-view diffusion transformation module is input into the next multi-view diffusion transformation module, and all views are traversed again one by one; or, The association between the first-view visual features and the current sparse implicit structure is established through a cross-attention mechanism, specifically as follows: By using sparse voxel features as query features and current view visual features and pose features as condition features, sparse voxels can extract geometric and appearance information related to their spatial position from the current view.

8. The method for generating 3D assets based on multi-view consistency constraints as described in claim 7, characterized in that, The optimized sparse implicit structure is geometrically and visually decoded to generate 3D assets, specifically including: The optimized sparse implicit structure is input into the Gaussian representation decoder to obtain a three-dimensional Gaussian representation of the target object; The optimized sparse implicit structure is input into the symbolic range field decoder to obtain the symbolic range field representation of the target object; Extract the 3D mesh of the target object based on the symbolic distance field representation; Multi-view rendering of the 3D Gaussian representation around the target object is performed to obtain an image sequence containing information about the surface appearance of the target object. The surface appearance information in the image sequence is mapped to the three-dimensional mesh to obtain a textured three-dimensional asset; or, A contour surface extraction algorithm is used to extract a three-dimensional mesh from the symbolic distance field representation.

9. The method for generating 3D assets based on multi-view consistency constraints as described in claim 8, characterized in that, When training the multi-view guided optimization network, a joint loss function is used for supervision. This joint loss function includes color reconstruction loss, structural similarity loss, perceptual loss, noise prediction loss, and geometric surface loss, where: The color reconstruction loss is used to measure the pixel color error between the generated or rendered image and the real image. The structural similarity loss is used to constrain the local structural consistency between the generated or rendered image and the real image; The perceptual loss is used to constrain the consistency between the generated or rendered image and the real image in the deep visual feature space. The geometric surface loss is used to constrain the surface consistency between the generated symbolic distance field or the three-dimensional surface and the geometry of the real object.

10. A 3D asset generation system based on multi-view consistency constraints, characterized in that, include: The reference image acquisition module is configured to acquire a single reference image containing the target object. The consistent multi-view generation module is configured to generate a set of multi-view images corresponding to the target object based on the single reference image and the preset target camera pose using a bidirectional recursive multi-view generation method. The coarse geometry proxy construction module is configured to: estimate the depth information and confidence information of each view based on the multi-view image set and the camera pose corresponding to each view image, project the effective pixels in each view to a unified three-dimensional coordinate system, and perform confidence filtering and spatial fusion on the projected three-dimensional points to obtain the coarse geometry proxy point cloud of the target object. The sparse implicit structure construction module is configured to: voxelize the coarse geometric proxy point cloud, determine the spatial location of the effective voxels, configure initial features for each effective voxel, and obtain the initial sparse implicit structure of the target object. The multi-view guided optimization module is configured to: input the initial sparse implicit structure, multi-view images and corresponding camera poses into the multi-view guided optimization network, and perform view-by-view iterative updates on the sparse implicit structure through the interaction between the multi-view image features and the sparse implicit structure to obtain the optimized sparse implicit structure. The 3D asset generation module is configured to perform geometric decoding and appearance decoding on the optimized sparse implicit structure to generate 3D assets.

Images