Free multi-view three-dimensional reconstruction method based on double-flow denoising and electronic device

Abstract

The application provides a free multi-view three-dimensional reconstruction method based on double-flow denoising and electronic equipment. The free multi-view three-dimensional reconstruction method based on double-flow denoising comprises the following steps: acquiring a sparse multi-view image of a target object; extracting a multi-view conditional feature from the sparse multi-view image; sampling an initial three-dimensional geometric latent variable and an initial view-by-view canonical point graph latent variable from a prior distribution to form an initial joint generation state; performing bidirectional information exchange between the three-dimensional geometric latent variable and each view-by-view canonical point graph latent variable, and obtaining a current joint generation state based on a generation time step; obtaining a denoised three-dimensional geometric latent variable and a canonical point graph latent variable of each view; and solving camera parameters corresponding to each input view based on three-dimensional canonical geometry and a view-by-view canonical point graph. The application achieves the purpose of establishing the spatial consistency of geometry and point graph without additional coordinate transformation, and improving the internal consistency of the generation result through continuous evolution of the state.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and in particular relates to a free multi-view method based on dual-stream denoising. Figure 3 Reconstruction methods and electronic equipment. Background Technology

[0002] In 3D static and dynamic modeling, recovering a 3D model from multiple images of the same target object is a fundamental task in computer vision and computer graphics. Traditional multi-view reconstruction typically involves first extracting and matching local features between different images, estimating camera intrinsic and extrinsic parameters based on the matching results, and then obtaining the 3D geometry through triangulation, multi-view stereo matching, and surface fusion. These methods rely on dense view coverage, sufficient image overlap, stable texture features, and relatively accurate camera parameters. When the number of input images is small, the camera pose is unknown, objects are occluded, or surface texture is insufficient, feature matching and camera estimation can easily become unstable, ultimately only recovering the observed surface portion from the input images.

[0003] Feedforward reconstruction methods can directly predict depth, point maps, and camera parameters from one or more images. Point maps are typically indexed by image pixels, storing the corresponding 3D point coordinates for each pixel, thus preserving pixel-level local observation information from the input image. These methods primarily recover geometry based on visible evidence in the input image, and their output is usually located in the camera coordinate system, the reference view coordinate system, or the reconstructed coordinate system determined by the model.

[0004] 3D generation methods utilize shape distributions learned on large-scale 3D assets to generate complete 3D objects within the latent variable distribution of the object's canonical space. The generation process can first produce a sparse 3D structure, and then form structured latent variables that carry detailed geometry and appearance. These methods can use complete shape priors to fill in unobserved regions, and their generation space needs to establish corresponding spatial relationships with the pixel space and camera space of the input image.

[0005] To simultaneously leverage the pixel alignment capabilities of feedforward reconstruction and the complete shape prior of 3D generation methods, existing fusion approaches typically introduce reconstruction features, point maps, camera poses, or pixel alignment features into the 3D generation process, and connect the feedforward reconstruction space and the 3D generation space through conditional injection, branch feature exchange, or spatial transformation.

[0006] In this type of fusion approach, such as Mix3R (a hybrid network of generative coarse geometry and feedforward point map prediction), the approach is designed for sparse pose-free multi-view inputs and processes 3D generation, per-view point map prediction, and camera spatial relationships within the same overall framework. This approach employs a two-stage coarse-to-fine workflow: the first stage jointly forms the coarse 3D structure, per-view point map, camera parameters, and spatial alignment relationships; the second stage utilizes the alignment relationships obtained in the first stage to generate detailed geometry and textures.

[0007] In the first stage, Mix3R sets up a 3D generation branch, a feedforward reconstruction branch, and a spatial transformation branch. The 3D generation branch is based on the TRELLIS sparse structure flow model, predicting the flow matching velocity for the latent variables of the sparse structure with added noise, to iteratively form a coarse 3D occupied structure. The feedforward reconstruction branch is based on π³, directly regressing the camera space point map and camera pose of each view from the features of the multi-view image. Since the π³ output and the TRELLIS voxel generation result are in different coordinate spaces, the spatial transformation branch further predicts the similarity transformation composed of scale, rotation, and translation, aligning the point map after camera parameter transformation to the TRELLIS voxel space.

[0008] Mix3R uses a hybrid transformer network to connect the 3D generation branch, the feedforward reconstruction branch, and the transformation branch. Some network layers inject feedforward reconstruction features into the 3D generation branch through cross-attention, while some global network layers perform joint attention calculations on 3D features, image features, and transformation features. This allows the 3D generation prior to influence point map prediction, while the reconstruction features constrain the generation of coarse 3D structures. During training, the 3D branch uses flow matching loss; the point map, camera, and spatial transformation are aligned and supervised using point coordinates and normal errors. The point map is decoded and output by the feedforward branch and is not set as a latent variable with independent noise addition states and independent denoising speed targets.

[0009] In the second stage, Mix3R decodes the coarse structure from the first stage into an occupant volume. Based on the aligned point map, it calculates the overlap between 3D voxels and each input image patch, converting this overlap into an attention bias, which is then incorporated into a pre-trained detailed geometry and texture generation model. This stage utilizes the alignment established in the first stage to control the position of the input texture on the final 3D object and can further optimize camera parameters through differentiable rendering.

[0010] In summary, Mix3R's overall technical organization includes generative coarse 3D structure branches, feedforward per-view point maps and camera prediction branches, similarity transformation branches for connecting different output spaces, and hybrid transformers for realizing multi-branch feature exchange.

[0011] In the process of implementing this application, the inventors discovered the following problems with the prior art: 1. The three-dimensional geometry and the point graph of each view adopt a heterogeneous state formation method, and the spatial consistency between the two depends on additional coordinate transformations.

[0012] 2. Two-way feature interaction has not yet formed the co-evolution of three-dimensional geometric state and point graph state in the complete generated trajectory. Summary of the Invention

[0013] To address the aforementioned problems in the existing technology, this invention provides a free multi-view based on dual-stream denoising. Figure 3 Reconstruction methods and electronic equipment.

[0014] In a first aspect, embodiments of this disclosure provide a free multi-view based on dual-stream denoising. Figure 3 3D reconstruction methods include: Obtain sparse multi-view images of the target object; Extract multi-view conditional features from the sparse multi-view image, the multi-view conditional features including cross-view fusion features and pixel alignment conditional features corresponding to each view; Initial three-dimensional geometric latent variables and initial view-by-view canonical point map latent variables are sampled from the prior distribution to form an initial joint generation state. The three-dimensional geometric latent variables are used to characterize the complete three-dimensional canonical geometry, and the view-by-view canonical point map latent variables are used to characterize the canonical point map corresponding to each input view, which is indexed by the two-dimensional pixel position of the image and records the three-dimensional coordinates of the visible surface points in the object canonical space. Based on the initial joint generation state, bidirectional information exchange is performed between the three-dimensional geometric latent variables and the latent variables of each view-by-view standardized point graph. Based on the generation time step, the three-dimensional geometric latent variables and the latent variables of each view-by-view standardized point graph are updated respectively to obtain the current joint generation state. In multiple generation time steps, the current joint generation state, temporal embedding, and multi-view conditional features are input into the joint generation network to obtain denoised 3D geometric latent variables and canonical point map latent variables for each view. The denoised 3D geometric latent variables are decoded into 3D normalized geometry, and the denoised view normalized point map latent variables are decoded into corresponding view-by-view normalized point maps. Based on the 3D normalized geometry and the view-by-view normalized point maps, the camera parameters corresponding to each input view are solved.

[0015] Optionally, the data of the three-dimensional normalized geometry and the view-by-view normalized point plot adopt the same object normalized space, so that the latent variables of the three-dimensional geometry and the latent variables of the view-by-view normalized point plot correspond to the same three-dimensional shape and normalized space assumptions during the joint generation process.

[0016] Optionally, the bidirectional information exchange between the three-dimensional geometric latent variables and the latent variables of each view-by-view specification point plot includes: The view-by-view standardized point plot latent variables are corrected using pixel-level local observation information, and the three-dimensional geometric latent variables are inversely corrected using complete shape priors and cross-view structural information, forming a bidirectional mutual correction throughout the entire process.

[0017] Optionally, the step of updating the three-dimensional geometric latent variables and the latent variables of each view-by-view canonical point plot based on the generation time step to obtain the current joint generation state includes: At each generation time step, the geometric feature labels corresponding to the three-dimensional geometric latent variables and the point map feature labels corresponding to each view-by-view standardized point map latent variables are combined into a joint sequence; Self-attention computation is performed on the joint sequence, enabling the geometric feature markers to read the pixel-level correspondence information in the point map feature markers, and simultaneously enabling the point map feature markers to read the complete three-dimensional shape assumptions and cross-view information in the geometric feature markers.

[0018] Optionally, inputting the current joint generation state, temporal embedding, and multi-view conditional features into the joint generation network includes: Global context is obtained by aggregating cross-view fusion features from the multi-view conditional features across attention; The dot plot feature markers are fused with the pixel alignment condition features of the corresponding view according to their spatial position to maintain local correspondence accuracy.

[0019] Optionally, extracting multi-view conditional features from the sparse multi-view image includes: A visual geometry fundamental transformer is used to extract geometric features for cross-view fusion; A self-supervised visual feature model is used to extract the semantic and local visual features of each view of the target object; By fusing the geometric features, semantic features, and local visual features, a visual representation of the target object is formed. Figure 1 One-to-one conditional features.

[0020] Optionally, the training steps of the joint generative network include: Add Gaussian noise to the latent variables of the 3D geometry and the latent variables of the standard point plot for each view; Randomly select the generation time, and according to the generation time, perturb the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point plot along the path from clean data to Gaussian noise to a noisy state; The noisy state, generation time, and multi-view conditional features are input into the joint generation network to simultaneously predict the update amount of the 3D geometric state and all view-by-view canonical point map states. The errors between the predicted 3D geometric state update and the corresponding target update, as well as the errors between the state update of each view-by-view specification point map and the corresponding target update, are calculated separately, and the joint generation network is jointly optimized according to preset weights.

[0021] Optionally, the method may also include a step of recovering the camera pose based on the decoded view-by-view canonical point map: Read the corresponding three-dimensional standard coordinates from the foreground pixel position of the view-by-view standard point map, and establish the correspondence between dense two-dimensional pixels and three-dimensional standard coordinates of the view corresponding to the view-by-view standard point map; By combining the camera intrinsic parameters of the target object, the camera pose is solved by perspective n points, so as to minimize the reprojection error between the pixel position obtained by the 3D normal coordinate projection and the corresponding 2D pixel position in the input view.

[0022] Optionally, the solution for the perspective n-points involves using random sampling consistency to filter inlier points and obtain initial poses, and then using a nonlinear optimization method to minimize the inlier point reprojection error.

[0023] Secondly, embodiments of this disclosure also provide an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the free multi-view based dual-stream denoising as described in any of the first aspects. Figure 3 Dimensional reconstruction method.

[0024] The present invention provides a free multi-view based on dual-stream denoising. Figure 3 The 3D reconstruction method performs bidirectional information exchange between the 3D geometric latent variables and the latent variables of each view-by-view standard point map. Based on the generation time step, it updates the 3D geometric latent variables and the latent variables of each view-by-view standard point map respectively to obtain the current joint generation state. In multiple generation time steps, the current joint generation state, temporal embedding, and multi-view conditional features are input into the joint generation network, thereby achieving the goal of establishing spatial consistency between geometry and point map without additional coordinate transformation, and improving the intrinsic consistency of the generation results through continuous state evolution. Attached Figure Description

[0025] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0026] Figure 1Free multi-view based on dual-stream denoising provided in embodiments of this disclosure Figure 3 Flowchart of the 3D reconstruction method; Figure 2 Free multi-view based on dual-stream denoising provided in embodiments of this disclosure Figure 3 Schematic diagram of the reconstruction device; Figure 3 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0029] It should be noted that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice method. Furthermore, this device and / or practice method can be implemented using other structures and / or functionalities besides one or more of the aspects set forth herein.

[0030] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0031] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0032] The normed space assumption refers to the fact that, during the joint generation process, the complete 3D normed geometry and the view-by-view normed point graph are both constrained to the same object normed coordinate system centered on the target object itself and normalized in scale and orientation.

[0033] The complete 3D shape assumption refers to the overall geometric configuration, located within the object's canonical space and encompassing the observed and unobserved areas of the target object, characterized by the decoding of the current 3D geometric latent variables at any time step in the joint generation process. This configuration is iteratively updated from the initial random latent variables with each generation time step, and is solidified into a complete 3D canonical geometry at the end of generation. During the iteration process, it receives pixel-level visible observation corrections from the view-by-view canonical point map state, and simultaneously provides cross-view overall structure and complete shape prior corrections to the view-by-view canonical point map state.

[0034] Perspective n-point is a geometric algorithm for solving camera extrinsic parameters (rotation R and translation t) from known 3D points and their 2D projection positions in the image.

[0035] This embodiment addresses the scenario of generating a complete 3D object from sparse, pose-free multi-view images of the same target object. Its target output includes complete 3D normed geometry representing the overall shape of the object and unobserved areas, as well as a view-by-view normed point map corresponding to each input view. The view-by-view normed point map uses the 2D pixel positions of the image as indices, and its effective pixel records the 3D coordinates of corresponding visible surface points in the object's normed space. This map carries the dense spatial correspondence information between multi-view pixel observations and the complete 3D object, and can be used to recover the camera pose of each input view relative to the generated 3D object. The specific technical problem to be solved in this embodiment is as follows: (1) How to model the complete three-dimensional standard geometry and the view-by-view standard point graph together as a generated state with their own iterative trajectories.

[0036] Existing fusion approaches for sparse poseless multi-view reconstruction and 3D generation typically utilize a 3D generation model to form complete geometry and a feedforward reconstruction model to form point maps, depth, or camera information. These two types of results employ different state formation and update methods, making it difficult for per-view pixel observations to participate as continuously evolving generative states in the formation process of a complete 3D object. Therefore, this embodiment requires establishing a joint conditional generation process that includes latent variables for 3D canonical geometry and latent variables for per-view canonical point maps. This ensures that both types of latent variables start from an initial state, are continuously passed and iteratively updated during the generation process, and ultimately form their respective final outputs.

[0037] (2) How to make the three-dimensional standard geometry generation state and the view-by-view standard point graph generation state perform bidirectional mutual correction throughout the same object standard space.

[0038] The 3D standardized geometry carries the complete shape prior and overall structural information across views, while the view-by-view standardized point map carries local observation information aligned with the pixels of each input image. To enable both types of information to work together on the generated result, this embodiment requires establishing a direct connection between the two types of generated states within the same object's standardized space, and repeatedly exchanging intermediate states throughout the joint generation process: the view-by-view standardized point map continuously corrects the local structure and spatial position of the 3D geometry, while simultaneously, the 3D standardized geometry reversely corrects the local 3D coordinates and cross-view information of the view-by-view standardized point map. Figure 1 To the point of being responsive.

[0039] The core technical problem to be solved in this embodiment is: how to achieve free multi-view denoising based on sparse dual-stream denoising. Figure 3 In the 3D reconstruction scene, complete 3D normalized geometry and view-by-view normalized point map are jointly generated. Through bidirectional mutual correction of the two types of generated states in the same object normalized space throughout the entire process, the complete shape prior and multi-view pixel observation promote each other, and finally form a spatially consistent complete 3D object and view-by-view normalized point map, and restore the camera pose consistent with the 3D object.

[0040] To address problem 1 in the background technology, the purpose of this embodiment is to establish a joint generation paradigm for complete 3D standard geometry and view-by-view standard point plots. The latent variables of the 3D geometry and the latent variables of each view's standard point plot together constitute the conditional generation state. The two types of clean targets are defined to reside in the same object standard space and each has a generation trajectory evolving from an initial random state to the target result. This ensures that the complete 3D geometry and view-by-view standard point plots generated each time correspond to the same 3D shape and standard space assumption, internalizing their spatial consistency into the joint generation process.

[0041] To address problem 2 in the background technology, the purpose of this embodiment is to establish a bidirectional mutual correction mechanism between the two types of generated states in the complete generated trajectory. In each generation stage, the view-by-view standardized point map state corrects the 3D geometric state using pixel-level local observations, while the 3D geometric state uses complete shape priors and cross-view overall structural information to reverse-correct the standardized point map state. The updated two types of intermediate states continue into subsequent generation stages, thereby achieving the co-evolution of the complete shape assumption and the multi-view pixel correspondence assumption.

[0042] The purpose of this embodiment is to jointly generate complete 3D normalized geometry and view-by-view normalized point maps belonging to the same object normalized space and the same spatial assumption for sparse poseless multi-view input. Through bidirectional mutual correction throughout the process, the intrinsic spatial consistency of the two is improved. Furthermore, the dense 2D-to-3D correspondence provided by the generated normalized point maps is used to recover the poses of each input view camera consistent with the generated 3D object. Specifically, this embodiment discloses a 3D reconstruction method based on dual-stream coupling denoising of 3D geometry and normalized point maps for free pose multi-view input. Figure 3 Dimensional reconstruction method.

[0043] like Figure 1 As shown, this embodiment discloses a free multi-view method based on dual-stream denoising. Figure 3 3D reconstruction methods include: Step S101: Obtain a sparse multi-view image of the target object; This embodiment uses N sparse multi-view images of the same target object as input, where N is an integer not less than 2, and does not require pre-providing camera extrinsic parameters for each input view. The input images may contain foreground masks; if the foreground mask is not directly provided, the target object region can be obtained using existing image segmentation methods.

[0044] Step S102: Extract multi-view conditional features from the sparse multi-view image, wherein the multi-view conditional features include cross-view fusion features and pixel alignment conditional features corresponding to each view; This embodiment extracts multi-view conditional features from all input images to guide joint generation. The conditional features include at least cross-view fusion information and per-view pixel alignment information: the cross-view fusion information is used to express the visible geometric relationships and object-level structure between multiple input views, and the per-view pixel alignment information is used to maintain the local spatial correspondence between each input image and its canonical point map.

[0045] In a specific application scenario, a Visual Geometry Grounded Transformer (VGGT) is used to extract geometric features for cross-view fusion, and a self-supervised visual feature model DINOv2 is used to extract semantic and local visual features of each input view. A conditional network fuses these two types of features to form a feature set for each input view. Figure 1 One-to-one conditional features. The geometry generation state can aggregate all view conditions to obtain the object-level context, while each view-by-view normed point map generation state receives the pixel alignment conditions of its corresponding view.

[0046] Step S103: Sample initial three-dimensional geometric latent variables and initial view-by-view normalized point map latent variables from the prior distribution to form an initial joint generation state, wherein the three-dimensional geometric latent variables are used to characterize the complete three-dimensional normalized geometry, and the view-by-view normalized point map latent variables are used to characterize the normalized point map corresponding to each input view, which is indexed by the two-dimensional pixel position of the image and records the three-dimensional coordinates of the visible surface points in the object's normalized space; The prior distribution refers to the probability distribution used to sample the initial three-dimensional geometric latent variables and the latent variables of each initial view-by-view specification point plot. Specifically, a Gaussian distribution with zero mean and unit covariance is adopted. This distribution is consistent with the Gaussian noise distribution added to the clean latent variables during the training phase, so that the generated trajectory can be iteratively updated from pure noise along the flow matching or diffusion path to the clean latent variables.

[0047] For the three-dimensional geometry of the target object, this embodiment first unifies its center, scale, and orientation into the object's normed space to obtain the three-dimensional normed geometry. In a specific application scenario, the object's normed space is normalized to a preset unified cube range, enabling different training objects to use a unified spatial range, and ensuring that the normed point maps of all input views use this three-dimensional normed geometry as a common reference.

[0048] Three-dimensional standard geometry is used to describe the complete surface and overall structure of a target object, including both the surface areas that can be directly observed in the input image and the unobserved areas that need to be filled in by prior knowledge through 3D generation.

[0049] For each input view, this embodiment constructs a corresponding canonical point map. The canonical point map has a two-dimensional grid structure adapted to the corresponding input image, and each valid foreground pixel position records the three-dimensional coordinates of the visible surface point observed by that pixel in the object canonical space. The three coordinate channels of the canonical point map sequentially represent the three coordinate directions in the object canonical space.

[0050] Therefore, the canonical point map retains the two-dimensional pixel index structure of the input image, but its stored content is three-dimensional coordinates in the object canonical space. The three-dimensional canonical geometry and all view-by-view canonical point maps are located in the same object canonical space in terms of data definition, so that image pixels, canonical point map coordinates, and generated geometry have a common three-dimensional reference.

[0051] During the training phase, canonical point map supervision can be constructed using training data with 3D models and rendering cameras. Specifically, the training object is transformed into the object canonical space, pixel rays are projected from each training view onto the object surface, the intersection points of the foreground pixel rays and the visible object surface are recorded, and the object canonical coordinates of the intersection points are written to the corresponding pixel positions. Background pixels, pixels without valid intersection points, and boundary pixels that need to be excluded can be removed from the canonical point map reconstruction loss and joint generation loss using foreground masks or invalid markers.

[0052] This embodiment employs a 3D geometric encoder to encode 3D standard geometry into compact geometric latent variables. In a specific application scenario, the 3D geometric encoder is the encoding part of a pre-trained 3D autoencoder, and the geometric latent variables are in the form of a set of feature labels, consisting of multiple geometric feature labels. This representation can be processed by a transformer-based generative network and carries the local and global 3D structural information of the complete object.

[0053] After generation, the 3D geometry decoder restores the generated geometric latent variables to complete 3D normal geometry.

[0054] This embodiment employs a standardized point map encoder to encode the standardized point maps of each input view separately. Each standardized point map latent variable maintains a two-dimensional mesh structure corresponding to its corresponding input view, and its spatial resolution can be lower than the original standardized point map resolution. All view-by-view standardized point map latent variables are organized according to the order of the input views and together with the three-dimensional geometric latent variables in the subsequent joint generation process to constitute the generated state.

[0055] In a specific application scenario, the canonical point map encoder and decoder employ a point map VAE fine-tuned from a pre-trained FLUX image model using a variational autoencoder (VAE). The input and output channels of the pre-trained image VAE, originally representing red, green, and blue colors, are replaced with channels representing the three canonical coordinate directions. The VAE is then trained using canonical coordinate reconstruction loss and Kullback-Leibler (KL) divergence regularization, ensuring that its latent variables preserve pixel-level coordinate information and local geometric details within the canonical point map. After generation, the canonical point map decoder decodes the latent point map variables of each view into the corresponding view-by-view canonical point map.

[0056] Step S104: Based on the initial joint generation state, perform bidirectional information exchange between the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point map, and update the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point map respectively based on the generation time step to obtain the current joint generation state; At each generation time step, this embodiment uses the current 3D geometric latent variables and all view-by-view canonical point plot latent variables as the joint generation state. The 3D geometric latent variables represent the current complete 3D geometric intermediate assumptions, and each canonical point plot latent variable represents the current canonical point plot intermediate assumptions for the corresponding view. Both types of states originate from their respective prior distributions at the start of generation and are continuously propagated and updated in subsequent generation stages.

[0057] In a specific application scenario, the geometry-point graph joint generation network employs a dual-branch diffusion transformer (DiT). The geometry branch processes 3D geometric feature markers, while the point graph branch processes canonical point graph feature markers from N views. Since the canonical point graph latent variables maintain a 2D mesh structure, the latent variables of each view's point graph are flattened into a 1D feature marker sequence before entering the shared interaction unit.

[0058] At each generation time step, a joint sequence is formed by combining temporal information, 3D geometric feature labels, and all flattened, view-by-view canonical point map feature labels. Shared self-attention computation is performed on this joint sequence, enabling the geometric feature labels to read pixel-level correspondence information from the view-by-view canonical point maps, and simultaneously enabling the view-by-view canonical point map feature labels to read complete 3D shape assumptions and cross-view information provided by other views. By repeatedly performing this shared self-attention across multiple network layers and multiple generation time steps, the two types of intermediate generated states can continuously correct each other.

[0059] After completing the bidirectional information exchange through shared self-attention, this embodiment injects multi-view conditions according to the different spatial structures of the two types of generated states.

[0060] For the 3D geometry branch, geometric feature labels represent the object-level 3D structure and do not have a fixed one-to-one pixel positional relationship with any single input image. Therefore, the geometry branch aggregates conditional features from multiple views across attention to obtain the global context of the complete object and obtain updated geometric latent variables.

[0061] For each canonical point map branch, its two-dimensional grid position maintains a correspondence with the local region of the corresponding input image. Therefore, the point map features are fused with the corresponding pixel alignment conditions according to their spatial positions, and then the updated point map features are obtained through feature projection.

[0062] This differential condition injection enables the 3D geometry branch to synthesize the overall evidence from multiple views, while maintaining the local pixel connections between each canonical point map branch and the corresponding input image.

[0063] After shared self-attention, modal differential condition injection, and feedforward networks for each branch, the geometry branch and the point graph branch predict the update amounts of the 3D geometric state and the view-by-view canonical point graph state, respectively, through the output projection head. These two types of update amounts are used together to update the joint generated state, allowing both the 3D geometric intermediate hypothesis and the view-by-view canonical point graph intermediate hypothesis to proceed to the next generation stage.

[0064] Step S105: In multiple generation time steps, input the current joint generation state, temporal embedding, and multi-view conditional features into the joint generation network to obtain the denoised 3D geometric latent variables and the canonical point map latent variables of each view; This embodiment employs a rectified flow-based flow matching training method. In each training sample, independent Gaussian noise is sampled for both the clean 3D geometric latent variables and the latent variables of each clean canonical point plot. A generation time is randomly selected, and each latent variable is perturbed along its path from clean data to Gaussian noise, resulting in a noisy state. The path can be a linear interpolation path or a path generated under other conditions.

[0065] The geometry-pointmap joint generation network receives joint noisy states, generation time, and multi-view conditions, and simultaneously predicts the update amounts of the 3D geometry state and all view-by-view canonical pointmap states. During training, the errors between the predicted geometry update amount and the corresponding target update amount, as well as the errors between each predicted pointmap update amount and the corresponding target update amount, are calculated separately; these two types of errors are jointly optimized according to preset relative weights. Through this joint training objective, the 3D geometry and view-by-view canonical pointmaps are jointly optimized in the same training samples, the same generation time, and the same joint network. The pixel-level supervision provided by the canonical pointmap branch acts on the 3D geometry branch through shared interactions, and the complete shape prior learned by the 3D geometry branch acts back on each view canonical pointmap branch through shared interactions.

[0066] In the training process of an application scenario, the canonical point map VAE is first trained or fine-tuned using the rendered canonical point map to accurately reconstruct the information of the three canonical coordinate directions; then the geometry and canonical point map encoder and decoder are fixed or partially fixed, and the bi-branch DiT and conditional network are trained to learn the joint conditional generation distribution of the 3D canonical geometry and the view-by-view canonical point map.

[0067] Step S106: Decode the denoised 3D geometric latent variables into 3D normalized geometry, and decode the denoised view normalized point map latent variables into corresponding view-by-view normalized point maps. Solve for the camera parameters corresponding to each input view based on the 3D normalized geometry and the view-by-view normalized point maps.

[0068] For three-dimensional geometric latent variables, the decoding part of a pre-trained three-dimensional autoencoder can be used as the geometric decoding module. This decoder takes the three-dimensional geometric latent variables as input for decoding.

[0069] For latent variables in a canonical dot plot, a canonical dot plot decoder can be used, which is a dot plot VAE decoder obtained by fine-tuning a pre-trained FLUX image VAE.

[0070] For each input view, the corresponding 3D canonical coordinates in the generated canonical point map are read from the foreground pixel position, thereby establishing the correspondence between the dense 2D pixels and the 3D canonical coordinates of the view.

[0071] By combining the camera intrinsic parameters of the corresponding input views, the camera pose is solved using Perspective-n-Point (PnP) to minimize the reprojection error between the pixel positions obtained from the 3D canonical coordinate projection and the corresponding pixel positions in the input image. To reduce the impact of occlusion boundaries, segmentation errors, or local outliers in the canonical point map, the perspective-n-points can be filtered using Random Sample Consensus (RANSAC) to obtain the initial pose, and then refined using the Levenberg-Marquardt method to minimize the inlier reprojection error. The final camera pose represents the position and orientation of each input view relative to the canonical space of the generated object.

[0072] In a specific application, only sparse multi-view images of the same target object need to be input; a real 3D model, a real canonical point map, or camera extrinsic parameters are not required. The inference process is executed in the following order: Step 1: Receive N sparse multi-view images and obtain the foreground region of the target object in each view.

[0073] Step 2: Obtain cross-view fusion features and pixel alignment condition features corresponding to each view through the multi-view condition feature extraction module.

[0074] Step 3: Sample the initial 3D geometric latent variables and the initial canonical point plot latent variables of each view from the Gaussian prior distribution to form the initial joint generation state.

[0075] Step 4: Based on the initial joint generation state, perform bidirectional information exchange between the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point map, and update the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point map based on the generation time step to obtain the current joint generation state; Step 5: At each generation time step, input the current joint state, temporal embedding, and multi-view conditions into the geometry-point graph joint generation network. Perform bidirectional information exchange through shared self-attention, and update the geometry state and the canonical point graph state of each view respectively through modal differential condition injection. Repeat Step 5 until the denoised 3D geometric latent variables and the canonical point graph latent variables of each view are obtained.

[0076] Step 6: Obtain the complete 3D standard geometry through the 3D geometry decoder, and obtain the view-by-view standard point map of each view through the standard point map decoder.

[0077] The above reasoning process allows both the 3D geometry and the canonical point map to start from their respective initial random states and continuously correct each other during the same sampling process, ultimately forming two types of directly generated results belonging to the canonical space of the same object and the same spatial assumption.

[0078] The technical means in this embodiment can be replaced by the following technical means: For the joint trajectory generation, this embodiment can employ rectified flow and linear probability paths, or a conditional denoising diffusion model to predict noise, clean samples, or equivalent updates for the 3D geometric state and the canonical point map state of each view at multiple denoising time steps. Alternatively, a conditional flow matching model with different probability paths or velocity parameterization methods can be used. A consensus model or consensus distillation method can also be employed to shorten the number of joint sampling steps, but in each joint update stage, the 3D geometric state and the canonical point map state still serve as both network input and output, maintaining their state interaction relationship.

[0079] The dual-branch diffusion converter in this embodiment is the preferred network organization. The geometric state and canonical point graph state can be processed by a multimodal diffusion converter (MMDiT), a multi-stream diffusion converter (MFDiT), or a converter network with a shared backbone and mode-specific input / output headers. Shared self-attention can also be replaced by bidirectional cross-attention alternating between the geometric branch and the canonical point graph branch, or by an interaction unit formed by a combination of a shared interaction layer and an intra-branch attention layer. Regardless of the network organization used, the alternative network should establish bidirectional information exchange between the two types of currently generated states and output the update results for both types of states separately, rather than simply treating the canonical point graph as a fixed condition or a one-time regression result.

[0080] Three-dimensional geometric latent variables can be represented using unordered point set feature labels, sparse voxel features, three-plane features, implicit field parameters, three-dimensional Gaussian parameters, or other compact representations that can be decoded into a complete three-dimensional object. The view-by-view canonical point map can directly use a three-channel canonical coordinate map, or it can be represented using grid latent variables, block feature labels, or multi-scale coordinate maps compressed by a two-dimensional autoencoder; foreground validity, normal, or confidence channels can also be added. Regardless of the representation used, the canonical point map should maintain the correspondence between the pixel positions of the input image and the three-dimensional coordinates of the visible surfaces in the canonical space of the same object, enabling it to participate in joint updates as a view-by-view generation state and providing a two-dimensional to three-dimensional correspondence after generation.

[0081] The visual geometry fundamental transformer and self-supervised visual feature model can also be replaced with other visual networks capable of extracting cross-view geometric relationships, object-level context, and view-by-view local visual features from sparse multi-view images. For conditional injection, the geometry branch can receive overall conditions for all views using cross-attention, conditional normalization, feature modulation, or feature aggregation; each canonical pointmap branch can receive pixel alignment conditions for the corresponding input view using stitching, position-by-position modulation, local cross-attention, or other methods that preserve view correspondence. This alternative preserves the differentiated conditional relationships between the geometric state utilizing overall evidence from multiple views and the canonical pointmap state utilizing local pixel evidence from the corresponding view.

[0082] The camera pose recovery in this embodiment can employ perspective n-point solving, random sampling consistency filtering, and Levonburg-Marquardt refinement. Alternatively, efficient perspective n-point solving, other minimal solvers, nonlinear optimization based on reprojection error, differentiable pose optimization, or combinations thereof, can be used to recover camera extrinsic parameters based on the 2D-to-3D correspondence provided by the generated canonical point map, and simultaneously estimate or optimize camera intrinsic parameters. This substitution does not change the core of this embodiment: the 2D-to-3D correspondence used in pose solving originates from a view-by-view canonical point map generated jointly with the complete 3D canonical geometry and located in the same object canonical space.

[0083] like Figure 2 As shown, this embodiment also discloses a free multi-view method based on dual-stream denoising. Figure 3 The reconstruction device includes the following functional modules: The module for constructing standardized spatial data representation and supervision is used to establish a common coordinate reference for standardized 3D geometry and standardized point plots for each view. The geometry and canonical point plot latent variable encoding module is used to convert 3D canonical geometry and view-by-view canonical point plots into clean latent variables during the training phase; The multi-view conditional feature extraction module is used to form cross-view global conditions and view-by-view pixel alignment conditions from the input image; The geometry-point graph joint generation module is used to construct two types of generation states and update them synchronously in the complete generation trajectory; The geometry and standard point plot decoding module is used to obtain complete 3D standard geometry and standard point plots of each view; The camera pose recovery module is used to solve the camera pose based on the dense two-dimensional-three-dimensional correspondence in the generated standard point map.

[0084] The multi-view images first enter the multi-view conditional feature extraction module; during the training phase, the normalized spatial data representation and supervised construction module establishes a common coordinate reference for the 3D normalized geometry and the view-by-view normalized point map; the geometry and normalized point map latent variable encoding module converts the 3D normalized geometry and normalized point map into clean latent variables by the corresponding encoders; the inference phase starts from two types of initial random latent variables; the geometry-point map joint generation module receives the two types of latent variables, generation time information, and multi-view conditions, and updates the geometry state and point map state simultaneously in each generation phase; after generation, the geometry and normalized point map decoding module outputs the 3D normalized geometry and the view-by-view normalized point map respectively; finally, the camera pose recovery module uses the normalized point map to establish the correspondence and solve for the pose of each view.

[0085] Compared to Mix3R's approach of using generative 3D states, feedforward point map prediction, and additional similarity transformations to align the two output spaces, this embodiment sets the latent variables of the complete 3D canonical geometry and the latent variables of the view-by-view canonical point map as associated generative variables, and ensures that the two types of clean targets reside in the same object canonical space from a data definition perspective. Therefore, the complete 3D geometry and the view-by-view canonical point map obtained each time through joint generation correspond to the same 3D shape and canonical space assumption. Their consistency is established by the joint generation distribution and the common space definition, which helps reduce the dependence on additional similarity transformations connecting the geometric generation space and the point map prediction space, and reduces the transmission of spatial transformation estimation bias between geometry, point map, and camera relationships.

[0086] In the joint generation process, this embodiment treats both 3D geometry and view-by-view canonical point maps as states that are continuously transmitted and updated across generation time, allowing the two types of intermediate hypotheses to be repeatedly exchanged and corrected throughout the complete generation trajectory. Simultaneously, the 3D geometric state aggregates multi-view overall conditions to obtain complete shape and cross-view structural information, while each view-by-view canonical point map state receives pixel alignment conditions to maintain local correspondence accuracy. This enables local pixel observations to continuously correct the 3D geometry, and allows the complete geometric prior to continuously correct the canonical point map in reverse. This co-evolution process is beneficial for simultaneously improving the consistency of the complete 3D shape, visible region pixel correspondence, and cross-view structure. The dense 2D and 3D correspondences established by the jointly generated canonical point map point to the same object canonical space as the currently generated 3D object, thus ensuring that the further recovered camera pose maintains the same spatial assumption as the generated geometry and canonical point map.

[0087] Object norm space refers to the three-dimensional coordinate space that is unified with respect to the object itself, including its center, scale, and orientation. Complete three-dimensional standard geometry refers to the three-dimensional result that represents the complete surface and overall structure of the target object in the object's standard space; A view-by-view standard point map is a coordinate graph that uses the two-dimensional pixel positions of the corresponding input image as indexes and records the three-dimensional coordinates of visible surface points in the standard space of an object at the effective pixel positions.

[0088] Standard point maps refer to view-by-view standard point maps, not ordinary depth maps, camera coordinate system point maps, or fixed conditions used only for one-time pose estimation.

[0089] The joint generated state refers to the state composed of the latent variables of the three-dimensional standard geometry and the latent variables of the standard point graph of all views, and is iteratively updated with the generation time; The bidirectional mutual correction throughout the entire process refers to the process in which the standard point map state is corrected by pixel-level local observations in multiple generation stages, and the three-dimensional geometric state is corrected by the complete shape prior and cross-view structural information. Both updated states are then passed on to subsequent generation stages.

[0090] Three-dimensional geometry and geometric state refer to the complete three-dimensional standard geometry and its generated state, respectively. Camera pose refers to the camera extrinsic parameters of the input view relative to the object's norm space.

[0091] This embodiment has the following effects: First, by incorporating complete 3D standard geometry and view-by-view standard point plots into the same joint generation framework, the spatial inconsistency between the two types of outputs is fundamentally eliminated.

[0092] Compared to existing technologies (such as Mix3R) that employ a heterogeneous combination of generative 3D state and feedforward point map predictions, and connect different output spaces through additional similarity transformations, this approach sets both the 3D geometric latent variables and the view-by-view canonical point map latent variables as relevant generative variables in the conditional generation process. Furthermore, both types of clean targets reside in the same object canonical space by data definition. Therefore, the complete 3D geometry and view-by-view canonical point map obtained from each joint generation naturally correspond to the same 3D shape and canonical space assumption. Their spatial consistency is intrinsically established by the joint generation distribution and common spatial definition, eliminating the need for post-hoc coordinate transformations for alignment. This reduces the risk of spatial transformation estimation bias propagating between geometry, point maps, and camera relationships.

[0093] Second, through a two-way mutual correction mechanism throughout the entire process, the co-evolution of complete shape prior and multi-view pixel observation is realized, which significantly improves the intrinsic consistency of the generated results.

[0094] Both 3D geometry and view-by-view specification point maps are continuously passed and updated across generation time steps, rather than treating point maps as one-time regression results or fixed conditions. At each generation time step, the geometric state can read pixel-level correspondence information from the view specification point maps, while the view specification point map states can read complete 3D shape assumptions and cross-view structural information. This bidirectional information exchange is repeatedly performed throughout the complete generation trajectory, allowing local pixel observations to continuously correct the local structure and spatial position of the 3D geometry, while the complete geometric prior continuously corrects the local 3D coordinates and cross-view information of the view specification point maps. Figure 1 Consistency. The two types of intermediate hypotheses gradually eliminate uncertainty in the co-evolution process, which is conducive to simultaneously improving the consistency of complete 3D shape, visible area pixel correspondence and cross-view structure.

[0095] Third, the jointly generated standard point map provides a dense 2D-3D correspondence with the 3D geometry for camera pose recovery, so that the pose solution results and the generated geometry maintain the same spatial assumptions.

[0096] Since the canonical point map and the complete 3D canonical geometry originate from the same joint generation process and both reside in the same object canonical space, the correspondence between dense 2D pixels and 3D canonical coordinates established by the canonical point map naturally points to the currently generated 3D object. Based on this correspondence, the camera pose obtained by solving for perspective n points belongs to the same spatial assumption as the generated geometry and the canonical point map, and can be directly used for subsequent texture baking, scene composition, or embodied intelligence tasks without additional spatial transformation alignment.

[0097] The electronic device disclosed in this embodiment includes a memory and a processor. The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0098] The processor may be a central processing unit (CPU) or other processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of this disclosure, the processor is used to execute computer-readable instructions stored in the memory, causing the electronic device to perform the free multi-view based dual-stream denoising of the foregoing embodiments of this disclosure. Figure 3 The reconstruction method includes all or part of the steps.

[0099] Those skilled in the art will understand that, in order to solve the technical problem of how to achieve a good user experience, this embodiment may also include well-known structures such as communication buses and interfaces, and these well-known structures should also be included within the protection scope of this disclosure.

[0100] like Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. It illustrates a structural schematic diagram suitable for implementing the electronic device in the embodiment of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0101] like Figure 3As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and data required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0102] Typically, the following devices can be connected to the I / O interface: input devices, such as sensors or visual information acquisition devices; output devices, such as displays; storage devices, such as magnetic tapes or hard drives; and communication devices. Communication devices allow electronic devices to exchange data wirelessly or via wired communication with other devices, such as edge computing devices. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0103] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the free multi-view based dual-stream denoising embodiments of this disclosure are performed. Figure 3 All or part of the steps of the dimensional reconstruction method.

[0104] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0105] A computer-readable storage medium according to embodiments of the present disclosure stores non-transitory computer-readable instructions thereon. When these non-transitory computer-readable instructions are executed by a processor, the free multi-view based dual-stream denoising of the foregoing embodiments of the present disclosure is performed. Figure 3 All or part of the steps of the dimensional reconstruction method.

[0106] The aforementioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or portable hard drive), media with built-in rewritable non-volatile memory (e.g., memory card), and media with built-in ROM (e.g., ROM cartridge).

[0107] For a detailed description of this embodiment, please refer to the corresponding descriptions in the foregoing embodiments, which will not be repeated here.

[0108] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0109] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0110] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0111] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0112] Various changes, substitutions, and modifications can be made to the techniques described herein without departing from the teachings defined in this embodiment. Furthermore, the scope of this embodiment is not limited to the specific aspects of the processes, machines, manufacturing processes, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufacturing processes, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, this embodiment includes such processes, machines, manufacturing processes, events, means, methods, or actions within its scope.

[0113] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0114] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A free-view 3D reconstruction method based on dual-stream denoising, characterized in that, include: Obtain sparse multi-view images of the target object; Extract multi-view conditional features from the sparse multi-view image, the multi-view conditional features including cross-view fusion features and pixel alignment conditional features corresponding to each view; Initial three-dimensional geometric latent variables and initial view-by-view canonical point map latent variables are sampled from the prior distribution to form an initial joint generation state. The three-dimensional geometric latent variables are used to characterize the complete three-dimensional canonical geometry, and the view-by-view canonical point map latent variables are used to characterize the canonical point map corresponding to each input view, which is indexed by the two-dimensional pixel position of the image and records the three-dimensional coordinates of the visible surface points in the object canonical space. Based on the initial joint generation state, bidirectional information exchange is performed between the three-dimensional geometric latent variables and the latent variables of each view-by-view standardized point graph. Based on the generation time step, the three-dimensional geometric latent variables and the latent variables of each view-by-view standardized point graph are updated respectively to obtain the current joint generation state. In multiple generation time steps, the current joint generation state, temporal embedding, and multi-view conditional features are input into the joint generation network to obtain denoised 3D geometric latent variables and canonical point map latent variables for each view. The denoised 3D geometric latent variables are decoded into 3D normalized geometry, and the denoised view normalized point map latent variables are decoded into corresponding view-by-view normalized point maps. Based on the 3D normalized geometry and the view-by-view normalized point maps, the camera parameters corresponding to each input view are solved.

2. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 1, characterized in that, The data of the three-dimensional normalized geometry and the view-by-view normalized point plot use the same object normalized space, so that the latent variables of the three-dimensional geometry and the latent variables of the view-by-view normalized point plot correspond to the same three-dimensional shape and normalized space assumptions during the joint generation process.

3. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 2, characterized in that, The bidirectional information exchange between the three-dimensional geometric latent variables and the latent variables of each view-by-view standardized point plot includes: The view-by-view standardized point plot latent variables are corrected using pixel-level local observation information, and the three-dimensional geometric latent variables are inversely corrected using complete shape priors and cross-view structural information, forming a bidirectional mutual correction throughout the entire process.

4. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 1, characterized in that, The process of updating the 3D geometric latent variables and the latent variables of each view-by-view canonical point graph based on the generation time step to obtain the current joint generation state includes: At each generation time step, the geometric feature labels corresponding to the three-dimensional geometric latent variables and the point map feature labels corresponding to each view-by-view standardized point map latent variables are combined into a joint sequence; Self-attention computation is performed on the joint sequence, enabling the geometric feature markers to read the pixel-level correspondence information in the point map feature markers, and simultaneously enabling the point map feature markers to read the complete three-dimensional shape assumptions and cross-view information in the geometric feature markers.

5. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 4, characterized in that, The step of inputting the current joint generation state, temporal embedding, and multi-view conditional features into the joint generation network includes: Global context is obtained by aggregating cross-view fusion features from the multi-view conditional features across attention; The dot plot feature markers are fused with the pixel alignment condition features of the corresponding view according to their spatial position to maintain local correspondence accuracy.

6. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 1, characterized in that, Extracting multi-view conditional features from the sparse multi-view image includes: A visual geometry fundamental transformer is used to extract geometric features for cross-view fusion; A self-supervised visual feature model is used to extract the semantic and local visual features of each view of the target object; By integrating the geometric features, semantic features, and local visual features, conditional features are formed that correspond one-to-one with each view of the target object.

7. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 1, characterized in that, The training steps for the joint generative network include: Add Gaussian noise to the latent variables of the 3D geometry and the latent variables of the standard point plot for each view; Randomly select the generation time, and according to the generation time, perturb the three-dimensional geometric latent variables and the latent variables of each view-by-view standard point plot along the path from clean data to Gaussian noise to a noisy state; The noisy state, generation time, and multi-view conditional features are input into the joint generation network to simultaneously predict the update amount of the 3D geometric state and all view-by-view canonical point map states. The errors between the predicted 3D geometric state update and the corresponding target update, as well as the errors between the state update of each view-by-view specification point map and the corresponding target update, are calculated separately, and the joint generation network is jointly optimized according to preset weights.

8. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 1, characterized in that, It also includes the step of recovering the camera pose from the view-by-view canonical point map based on decoding: Read the corresponding three-dimensional standard coordinates from the foreground pixel position of the view-by-view standard point map, and establish the correspondence between dense two-dimensional pixels and three-dimensional standard coordinates of the view corresponding to the view-by-view standard point map; By combining the camera intrinsic parameters of the target object, the camera pose is solved by perspective n points, so as to minimize the reprojection error between the pixel position obtained by the 3D normal coordinate projection and the corresponding 2D pixel position in the input view.

9. The free multi-view 3D reconstruction method based on dual-stream denoising according to claim 8, characterized in that, The solution for the perspective n-point problem involves using random sampling consistency to filter interior points and obtain initial poses, and then using a nonlinear optimization method to minimize the interior point reprojection error.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the free multi-view 3D reconstruction method based on dual-stream denoising as described in any one of claims 1-9.

Images