Dual-branch color point cloud 3D texture mesh reconstruction method, device and electronic equipment

Abstract

The present application belongs to the technical field of image processing, and provides a double-branch color point cloud 3D texture grid reconstruction method, device and electronic equipment, which comprises: performing multi-view projection and grid reconstruction on the input color point cloud respectively to obtain multi-view sparse images and non-texture 3D grid; performing multi-view consistency repair on the multi-view sparse images to generate an initial UV texture map; performing texture repair on the initial UV texture map based on back projection and weighted mixing algorithm to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and pasting the final UV texture map back to the surface of the non-texture 3D grid to generate a 3D texture grid. The scheme cooperatively optimizes the image space and the texture space through a double-branch framework, uses multi-view consistency repair to improve the texture definition and global consistency, and improves the definition of the texture map through texture repair and texture resolution enhancement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, and electronic device for reconstructing a dual-branch color point cloud 3D texture mesh. Background Technology

[0002] In recent years, with the rapid development of technologies such as virtual reality, augmented reality, digital twins, and metaverse, the reconstruction of 3D textured meshes based on colored point clouds has received widespread attention in the fields of computer vision and graphics. As a 3D data representation containing geometric coordinates and color information, colored point clouds can be transformed into textured 3D mesh models through appropriate processing, and are widely used in scenarios such as cultural heritage protection, 3D modeling, and real-time rendering.

[0003] In related technologies, one type of method achieves texture reconstruction by learning implicit representations of color information in 3D or UV space. These methods typically require large amounts of 3D data, but the acquisition of 3D data is costly and limited in quantity, resulting in poor generalization ability when dealing with unseen objects or complex scenes. Furthermore, because implicit representations struggle to accurately capture complex geometric details, the reconstruction results often suffer from texture blurring, especially at occluded boundaries or in high-frequency texture regions. Another type of method employs traditional surface reconstruction techniques, such as Poisson surface reconstruction, which directly generates textured meshes by fusing point cloud color information. However, this method relies on accurate point normal estimation, which can easily lead to geometric redundancy or color distortion when dealing with sparse point clouds or complex topologies, making it difficult to meet the requirements for high-quality textured mesh construction.

[0004] Meanwhile, significant progress has been made in image super-resolution and generation techniques in the field of 2D vision, especially with the emergence of diffusion models, which provide strong prior support for image inpainting and generation. Compared to 3D data, 2D image data is abundant and easily accessible, and the technology is more mature. However, introducing 2D diffusion models into 3D generation tasks still mainly involves predicting or optimizing color information in 3D space, which is time-consuming and difficult to eliminate texture blurring. In addition, existing diffusion models mostly use single-view images as input, and lack consideration of other viewpoint information when processing multi-view images, resulting in insufficient global consistency and often causing problems such as occlusion boundary artifacts, boundary texture misalignment, "seam" phenomena, and repetition of multifaceted features.

[0005] Therefore, traditional methods for reconstructing texture meshes from color point clouds suffer from technical problems such as low texture clarity, poor global consistency, and insufficient generalization ability. Summary of the Invention

[0006] This invention provides a method, apparatus, and electronic device for reconstructing 3D texture meshes from dual-branch color point clouds, which solves the shortcomings of traditional schemes for reconstructing texture meshes from color point clouds, such as low texture clarity, poor global consistency, and insufficient generalization ability.

[0007] On one hand, the present invention provides a method for reconstructing a dual-branch color point cloud 3D texture mesh, comprising:

[0008] Multi-view projection and mesh reconstruction are performed on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh.

[0009] Multi-view consistency restoration is performed on the multi-view sparse image to generate an initial UV texture map;

[0010] Based on back projection and weighted blending algorithms, texture restoration is performed on the initial UV texture map to generate a complete UV texture map;

[0011] The complete UV texture map is subjected to texture resolution enhancement processing to obtain the final UV texture map;

[0012] The final UV texture map is then applied back to the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0013] The dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention performs multi-view projection on the input color point cloud to obtain a multi-view sparse image, including:

[0014] For each viewpoint, based on preset camera parameters, the color point cloud is transformed to the camera coordinate system, and visible points on the convex hull are extracted to obtain a subset of visible point clouds for each viewpoint.

[0015] The visible points in the visible point cloud subset are projected onto a two-dimensional image plane, the pixel coordinates corresponding to each visible point are calculated, and the color is filled at the corresponding pixel coordinate position according to the color value of each visible point to generate a sparse image under each viewpoint.

[0016] Treat sparse images from all perspectives as multi-view sparse images.

[0017] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multi-view consistency restoration is performed on the multi-view sparse image to generate an initial UV texture map, including:

[0018] Construct a 2D diffusion model based on multi-view consistency;

[0019] The multi-view sparse image and preset key calculation parameters are input into the 2D diffusion model based on multi-view consistency, so as to generate multiple target images with consistent viewpoints through reverse diffusion in the 2D diffusion model based on multi-view consistency.

[0020] The multi-view information from the multiple target images with the same viewpoint is fused into the texture space to obtain the initial UV texture map.

[0021] The dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention generates multiple viewpoint-consistent target images through reverse diffusion in the multi-viewpoint-consistent 2D diffusion model, including:

[0022] Based on the multi-view sparse image and preset key calculation parameters, a noisy image is initialized for each view, and a degradation path from the original image to the noisy image is defined through a forward diffusion process to establish a multi-view joint probability distribution.

[0023] Starting from a completely noisy state, reverse diffusion is performed step by step according to the multi-view joint probability distribution. As the number of reverse diffusion steps decreases, image details are gradually restored, while the multi-view consistency constraint is continuously strengthened, and finally multiple target images with consistent viewpoints are output.

[0024] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multi-view information from multiple target images with consistent viewpoints is fused into a texture space to obtain an initial UV texture map, including:

[0025] Perform backprojection on the target image from each viewpoint to obtain the backprojection result for each viewpoint;

[0026] The back-projection results from each viewpoint are blended into a single UV map using a weighted average algorithm based on the incident angle, resulting in an initial UV texture map.

[0027] The dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, based on back projection and weighted mixing algorithms, performs texture repair on the initial UV texture map to generate a complete UV texture map, including:

[0028] Position maps and normal maps from multiple viewpoints are rendered from the textureless 3D mesh;

[0029] Based on the location map and normal map, a location texture map and a normal texture map are generated through back projection and weighted blending algorithms;

[0030] The initial UV texture map, the position texture map, and the normal texture map are all input into the pre-built texture restoration model to obtain a complete UV texture map.

[0031] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, the complete UV texture map is subjected to texture resolution enhancement processing to obtain the final UV texture map, including:

[0032] The complete UV texture map is divided into multiple overlapping patches;

[0033] Each overlapping patch was subjected to resolution enhancement and noise reduction to obtain multiple optimized patches;

[0034] Gaussian weighted fusion is performed on the overlapping regions of the multiple optimization patches to obtain the fusion result;

[0035] The fusion result is then subjected to global color correction and detail sharpening post-processing to obtain the final UV texture map.

[0036] The dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention reconstructs the input color point cloud into a textureless 3D mesh, including:

[0037] The colored point cloud is input into a pre-constructed POPO surface reconstruction model. The geometric features in the colored point cloud are extracted and the color information is separated through the POPO surface reconstruction model. The surface normal is predicted using point basis representation and object prior knowledge. After surface interpolation, meshing and optimization processing, a textureless 3D mesh is obtained.

[0038] On the other hand, the present invention also provides a dual-branch color point cloud 3D texture mesh reconstruction device, comprising:

[0039] The point cloud processing module is used to perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and textureless 3D meshes.

[0040] The consistency repair module is used to perform multi-view consistency repair on the multi-view sparse image and generate an initial UV texture map.

[0041] The texture restoration module is used to perform texture restoration on the initial UV texture map based on back projection and weighted blending algorithms to generate a complete UV texture map.

[0042] The enhancement module is used to perform texture resolution enhancement processing on the complete UV texture map to obtain the final UV texture map;

[0043] The generation module is used to apply the final UV texture map back to the surface of the textureless 3D mesh to generate a 3D textured mesh.

[0044] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dual-branch color point cloud 3D texture mesh reconstruction method as described above.

[0045] This invention provides a dual-branch color point cloud 3D texture mesh reconstruction method, apparatus, and electronic device. It obtains a multi-view sparse image and a textureless 3D mesh by performing multi-view projection and mesh reconstruction on the input color point cloud. The multi-view sparse image undergoes multi-view consistency restoration to generate an initial UV texture map. Based on back-projection and weighted mixing algorithms, the initial UV texture map is inlaid to generate a complete UV texture map. The complete UV texture map undergoes texture resolution enhancement to obtain a final UV texture map. The final UV texture map is then applied back onto the surface of the textureless 3D mesh to generate a 3D texture mesh. This scheme achieves collaborative optimization in image and texture spaces through a dual-branch framework. Utilizing multi-view consistency restoration, it improves texture clarity and global consistency. Furthermore, through texture restoration and texture resolution enhancement, it enhances the clarity of the texture map, thus adapting to different task requirements. This effectively solves the problems of low texture clarity, poor global consistency, and insufficient generalization ability in traditional schemes for reconstructing texture meshes from color point clouds. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment of the invention;

[0048] Figure 2 This is a schematic diagram illustrating the implementation principle of the dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment of the invention;

[0049] Figure 3 This is a schematic diagram illustrating the implementation principle of a 2D diffusion model based on multi-view consistency.

[0050] Figure 4 This is a comparison chart of the results of image space multi-view consistency optimization;

[0051] Figure 5 This is a comparison chart showing the results of texture space optimization and resolution enhancement.

[0052] Figure 6 This is a visual diagram showing the comparison between the method provided in this embodiment and the traditional method;

[0053] Figure 7 This is a schematic diagram of the structure of the dual-branch color point cloud 3D texture mesh reconstruction device provided in an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The following is combined Figures 1 to 8 This invention describes in detail the dual-branch color point cloud 3D texture mesh reconstruction method, apparatus, and electronic device provided in the embodiments of the present invention.

[0057] like Figure 1 As shown, the dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment of the invention mainly includes the following steps:

[0058] Step 110: Perform multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh.

[0059] Step 120: Perform multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map.

[0060] It should be noted that UV mainly refers to the UV coordinate system. In the UV coordinate system, the texture image can be regarded as a 2D plane, U corresponds to the horizontal direction, similar to the X-axis of the Cartesian coordinate system, and V corresponds to the vertical direction, similar to the Y-axis of the Cartesian coordinate system.

[0061] Step 130: Based on the back projection and weighted blending algorithm, perform texture repair on the initial UV texture map to generate a complete UV texture map.

[0062] Step 140: Perform texture resolution enhancement processing on the complete UV texture map to obtain the final UV texture map.

[0063] Step 150: Apply the final UV texture map back to the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0064] The dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment can be found in [reference needed]. Figure 2This reconstruction method can be executed in both the image space module 210 and the texture space module 220. The image space module 210 includes a multi-view projection and consistency restoration stage, while the texture space module 220 includes a texture restoration and texture resolution enhancement stage. In the image space module 210, an initial UV texture map can be generated using an image restoration diffusion model (i.e., a 2D diffusion model for multi-view consistency). In the texture space module 220, texture details can be optimized using a texture restoration diffusion model (i.e., a texture restoration model), and resolution can be optionally enhanced to generate the final texture map. Finally, the final texture map is applied back onto the untextured mesh to generate a high-quality 3D textured mesh.

[0065] In one embodiment, multi-view projection is performed on the input color point cloud to obtain a multi-view sparse image, including:

[0066] First, for each viewpoint, based on preset camera parameters, the color point cloud is transformed to the camera coordinate system, and visible points on the convex hull are extracted to obtain a subset of visible point clouds for each viewpoint.

[0067] Then, the visible points in the visible point cloud subset are projected onto the two-dimensional image plane, the pixel coordinates corresponding to each visible point are calculated, and the color is filled at the corresponding pixel coordinate position according to the color value of each visible point to generate a sparse image under each viewpoint.

[0068] Finally, the sparse images from all perspectives are treated as multi-view sparse images.

[0069] This embodiment can process the input color point cloud. S Perform multi-view projection to generate M Sparse images from various perspectives, among which... M The preset number of viewpoints is used. The color point cloud contains the 3D coordinates and RGB color values ​​of each point. First, to avoid occlusion points in the projection, this embodiment employs a hidden point removal operation. Specifically, for each viewpoint, based on its camera parameters, the color point cloud is transformed to the camera coordinate system, and visible points on the convex hull are extracted to obtain a subset of the visible point cloud for that viewpoint. Subsequently, the visible points are projected onto the 2D image plane, the pixel coordinates corresponding to each visible point are calculated, and color is filled at the corresponding pixel coordinate positions according to the color values ​​of the visible points to generate a sparse image for that viewpoint. Repeating this process can generate... M Zhang sparse image In practical applications, the viewing angle adopts a Fibonacci spherical distribution to ensure uniform coverage.

[0070] In one embodiment, multi-view consistency restoration is performed on a multi-view sparse image to generate an initial UV texture map, specifically including:

[0071] The first step is to construct a 2D diffusion model based on multi-view consistency.

[0072] Understandably, this is a novel zero-lens framework for arbitrary image restoration problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring, based on existing denoising diffusion null-space models. Specifically, the denoising diffusion null-space model is implemented based on a denoising diffusion probabilistic model. It progressively adds noise to the data through a forward process and reconstructs the target image from the noise through a reverse process. It utilizes range-null-space decomposition to ensure data consistency and generates texture details that conform to the natural image distribution by iteratively optimizing the null-space content.

[0073] Building upon this foundation, this invention analyzes the impact of multi-view consistency on texture reconstruction for sparse images generated by multi-view projection of color point clouds. For example, inconsistencies in color and boundaries between different views can lead to texture blurring or seam artifacts, while an insufficient number of views may result in the loss of global details. Combining multi-view consistency constraints, this embodiment proposes a 2D diffusion model based on multi-view consistency for texture generation. By jointly optimizing the multi-view sparse image obtained from the projection, it ensures that the generated initial UV texture map achieves good results in both global consistency and detail clarity.

[0074] The second step involves inputting the multi-view sparse image and preset key computational parameters into a 2D diffusion model based on multi-view consistency, so as to generate multiple target images with consistent viewpoints through reverse diffusion in the 2D diffusion model based on multi-view consistency.

[0075] In one specific implementation, multiple viewpoint-consistent target images are generated through reverse diffusion in a multi-viewpoint-consistent 2D diffusion model, specifically including:

[0076] First, based on multi-view sparse images and preset key computational parameters, a noisy image is initialized for each view, and a degradation path from the original image to the noisy image is defined through a forward diffusion process to establish a multi-view joint probability distribution.

[0077] Then, starting from a completely noisy state, reverse diffusion is performed step by step based on the joint probability distribution of multiple views. As the number of reverse diffusion steps decreases, image details are gradually restored, while the consistency constraint of multiple views is continuously strengthened, and finally multiple target images with consistent views are output.

[0078] In practical applications, such as Figure 3 As shown, the first step is to define and initialize the model input. The model input mainly includes... M Zhang sparse image And the corresponding key calculation parameters, specifically, the key calculation parameters include: camera pose and degradation factor. A mWhere m = 1, 2, ..., M. Simultaneously, initialize noisy images from M viewpoints. T represents the number of steps in the entire diffusion process, and t represents each step of the reverse diffusion.

[0079] Next, model the joint probability distribution of images from M viewpoints. M uniform dense images are generated through a single reverse diffusion. The forward process adds noise independently for each viewpoint, defining a degradation path from a sparse image to a noisy image, as shown in the following formula:

[0080] (1)

[0081] (2)

[0082] in, It is the probability of multi-perspective joint forward diffusion. It is a set of original images from M perspectives. It is a set of noisy images from M perspectives after diffusion from step 1 to step T. It is the probability distribution of diffusion at step t from a single perspective. These are the noise scheduling parameters for time step t. It is the noisy image at time step t from the m-th viewpoint. The reverse diffusion process is defined as:

[0083] (3)

[0084] (4)

[0085] in, It is the joint probability of back diffusion. It is a complete sequence of M viewpoints from the original image to the final noisy image. It is the initial noise distribution. It is the mean of the m-th viewpoint, depending on the intermediate state of all viewpoints. The calculation formula is:

[0086] (5)

[0087] in, , , The noise scheduling parameters are for time step t. is the noise predictor for the m-th viewpoint.

[0088] Then, in each step of the reverse diffusion, to ensure multi-view Figure 1 To ensure consistency, the model employs a 3D perceptual feature attention mechanism to synchronize features from M viewpoints. First, it starts from the intermediate states of the M viewpoints... Extract features and construct a A spatial feature stereomicroscope is constructed, where V is the resolution of the spatial feature stereomicroscope. Then, for each viewpoint m, a frustum stereomicroscope is constructed based on its camera pose, and features are extracted from the spatial feature stereomicroscope through interpolation. In the noise predictor... On the intermediate feature map, a depth-oriented attention mechanism is applied to extract features along the depth dimension of the view cone, ensuring multi-view... Figure 1 To the point of being responsive.

[0089] Furthermore, to further ensure data consistency in the generated images, the model employs range-null space decomposition. Specifically, for the noise predictor... Estimated dense image Application Scope - Spatial decomposition for data consistency correction, the specific formula is as follows:

[0090] (6)

[0091] in, For degenerate operators The false rebellion, Ensure that the range space satisfies data consistency This represents the null space. Then, the corrected... Switch back to the noise state for the next step of denoising.

[0092] After T-step reverse diffusion, the model generates M target images with consistent multi-view perspectives. These images not only satisfy data consistency but also maintain geometric and color consistency in 3D space, and can be directly used for subsequent 3D reconstruction or other multi-view applications. Figure 4 It is a visualization result that is consistent across multiple perspectives, using (i.e., part a) and not using (i.e., part b).

[0093] The third step is to fuse the multi-view information from multiple target images with the same viewpoint into the texture space to obtain the initial UV texture map.

[0094] After obtaining a target image with consistent multi-view perspectives, it is necessary to further fuse the multi-view information in the target image into the texture space to generate an initial UV texture map. The specific process includes back projection and weighted mixing operations based on the incident angle.

[0095] In one specific implementation, multi-view information from multiple target images with consistent viewpoints is fused into the texture space to obtain an initial UV texture map, specifically including:

[0096] First, perform a backprojection operation on the target image at each viewpoint to obtain the backprojection result at each viewpoint.

[0097] In this embodiment, for each non-background pixel in the target image, its corresponding grid surface point is determined, and its value is mapped to the corresponding coordinates in UV space. The specific formula is as follows:

[0098] (7)

[0099] in, K represents the pixel coordinates in the 2D image, R represents the intrinsic parameter matrix, and R and T both represent the camera extrinsic parameter matrices. Representing rays and meshes Calculation of the intersection point This indicates the back projection operation. Represents the coordinates in UV space.

[0100] Then, the backprojection results from each viewpoint are blended into a single UV map using a weighted average algorithm based on the incident angle, resulting in the initial UV texture map.

[0101] This embodiment optimizes the fusion quality by using a weighted average of the incident angles to blend the back projections into a single UV map. Specifically, the cosine similarity between the viewpoint direction and the pixel-by-pixel normal vector in the image space is used to determine the weight contribution of each pixel to the blended texture. The specific formula for the weighted average is as follows:

[0102] (8)

[0103] in, This represents the color value of pixel p in the merged UV texture map. This represents the image from the i-th viewpoint. The middle pixel p is back-projected onto the UV texture map. Represents the image of pixel p at the i-th viewpoint. The angle of incidence in The index represents the weighted average and is used to control the degree to which the angle of incidence affects the weights. This represents a very small constant to avoid the denominator being zero. Specifically, It can be designed as 6, incident angle The definition is as follows:

[0104] (9)

[0105] in, It is a vector and The angle between them It is the viewing direction from the camera's viewpoint i to pixel p. It is the normal vector at pixel p.

[0106] In one embodiment, mesh reconstruction is performed on the input color point cloud to obtain a textureless 3D mesh, specifically including:

[0107] The colored electric cloud is input into the pre-built POPO surface reconstruction model. The geometric features in the colored point cloud are extracted and the color information is separated through the POPO surface reconstruction model. The surface normal is predicted by using point basis representation and object prior knowledge. After surface interpolation, meshing and optimization processing, a textureless 3D mesh is obtained.

[0108] This embodiment uses a colored point cloud. S As input, a textureless 3D mesh is generated using the preprocessed color point cloud from the POPO (Point-based Object Priors) surface reconstruction model. The specific network structure of the model is as follows:

[0109] (10)

[0110] in, This represents the implicit surface representation at the query point. This represents the i-th point in the colored point cloud. The nearest neighbor set is represented, and local geometric relationships are calculated based on learned difference weights. This represents the point cloud convolution operation, and the specific operation is as follows:

[0111] (11)

[0112] in, for The input features are Conv, which is the point cloud convolution operation.

[0113] In one embodiment, texture inpainting is performed on the initial UV texture map based on a back-projection and weighted blending algorithm to generate a complete UV texture map, including:

[0114] The first step is to render position maps and normal maps from multiple viewpoints from a textureless 3D mesh.

[0115] The second step involves generating position texture maps and normal texture maps based on the position atlas and normal atlas using back projection and weighted blending algorithms.

[0116] In this embodiment, we first start with a textureless 3D mesh. The position atlas of M viewpoints is obtained through rendering. and normal atlas Then, using back projection and a weighted mixing operation based on the incident angle, the location atlas is... and normal atlas Blend into texture space to generate corresponding location texture maps. and normal texture map This is used as a subsequent common input to the texture restoration model to achieve geometry-guided texture detail restoration.

[0117] The third step involves inputting the initial UV texture map, position texture map, and normal texture map into the pre-built texture restoration model to obtain a complete UV texture map.

[0118] Because the initial UV texture map is generated simply by using back projection and a weighted blending operation based on the incident angle. The initial view selection may not adequately cover the 3D shape structure, leading to occlusion and missing texture maps. Additionally, when image pixels are back-projected into UV space, the relationship is not one-to-one, potentially causing pixel-level "holes" in the texture map. To generate a complete UV texture map... In this embodiment, the initial UV texture map is... With positional texture mapping and normal texture map A common input texture restoration model is used to achieve geometry-guided texture detail restoration.

[0119] Specifically, the texture inpainting model is a latent diffusion model based on U-Net, specifically a texture inpainting diffusion network obtained by fine-tuning the Emu model, denoted as... The specific structure of the texture restoration model is shown below:

[0120] (12)

[0121] in, As part of the input parameters for the texture restoration model, at the same time z It is a 2D noise map, where each pixel is independently sampled from a standard Gaussian distribution. It is the output of the model, which will contain a complete UV texture map. Loss calculations were performed on the real texture map to optimize the diffusion network.

[0122] This embodiment generates a textureless 3D mesh using mesh reconstruction technology, optimizes the initial UV texture map using a texture inpainting model based on back projection and weighted blending algorithms, and generates a complete UV texture map. The processing results for each texture space optimization and enhancement step, as well as the result after applying both, are detailed below. Figure 5 As shown.

[0123] In one embodiment, the complete UV texture map is subjected to texture resolution enhancement processing to obtain the final UV texture map, specifically including:

[0124] The first step is to divide the complete UV texture map into multiple overlapping patches.

[0125] The second step involves performing resolution enhancement and noise reduction on each overlapping patch to obtain multiple optimized patches.

[0126] This embodiment enhances the complete UV texture map using a patch-based approach. In practical applications, the resolution and quality of the target texture map can be adjusted according to different task requirements to generate target texture maps with corresponding resolutions. Specifically, the complete UV texture map is then used. The structure is divided into multiple overlapping patches, and a Multi Diffusion model is used as a pre-trained diffusion model. Multi Diffusion is a multimodal diffusion model library whose core idea is to use the diffusion process to gradually build complex structures, simulating the interaction between different modalities in reality. In this embodiment, the Multi Diffusion model is used to enhance the resolution of each patch, generating a high-resolution output through a denoising process. During this process, the location texture mapping... and normal texture map As a geometric guiding condition.

[0127] The third step is to perform Gaussian weighted fusion on the overlapping areas of multiple optimization patches to obtain the fusion result.

[0128] In this embodiment, overlapping areas are fused using Gaussian weighted fusion to avoid seams and inconsistencies.

[0129] The fourth step is to perform global color correction and detail sharpening post-processing on the fusion result to obtain the final UV texture map.

[0130] In practical applications, color correction can be achieved through histogram matching or global brightness adjustment, thus maintaining the consistency of texture color information with the original point cloud. Detail sharpening can be achieved by enhancing texture edges and surface details through high-frequency filtering. The generated high-resolution final UV texture map retains the clarity of local details while maintaining global consistency with the geometry and semantics of the 3D object, making it suitable for high-quality rendering, virtual reality, and other application scenarios.

[0131] This embodiment improves the resolution of the complete UV texture map by introducing an optional texture enhancement network, and maps the final UV texture map onto a surface without texture mesh to generate a 3D texture mesh.

[0132] To verify the processing effect of the dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment, this embodiment uses traditional methods such as SPR, NKSR, and Texture Field, as well as the method provided in this embodiment (denoted as Ours), to process the same object samples to be reconstructed (denoted as Ground Truth, i.e., real information) composed of different types of objects such as packaging boxes, backpacks, toy cars, and clocks. The visualization comparison results are as follows: Figure 6 As shown.

[0133] See Figure 6 SPR (Single-View Physically-Based Rendering), a traditional 3D reconstruction method, has significant flaws in the reconstruction results of the aforementioned object samples, such as blurred clock surfaces and lost backpack textures. NKSR (Neural Kernel Surface Reconstruction), a reconstruction method based on neural implicit representations, performs better than SPR, but still suffers from detail loss, such as missing toy car colors and clock textures. TextureField, a method focused on texture reconstruction, shows that some objects (such as packaging boxes) lose their original patterns, and the clock is almost entirely blank, indicating insufficient handling of complex textures. The method provided in this embodiment (i.e....) Figure 6 The reconstruction of Ours in the image is closer to the real information (i.e., ... Figure 6 In the Ground Truth, the textures and shapes of objects are preserved intact, such as backpack patterns and clock numerals, which are clearer.

[0134] In summary, compared with traditional processing methods, the method provided in this embodiment not only enhances the clarity of mesh textures but also effectively solves boundary misalignment and seam problems, maintaining the global consistency of 3D objects. Furthermore, it reduces dependence on 3D training data, improves generalization ability, and is applicable to various 3D reconstruction scenarios.

[0135] Based on the same general inventive concept, this invention also protects a dual-branch color point cloud 3D texture mesh reconstruction device. The dual-branch color point cloud 3D texture mesh reconstruction device provided by this invention will be described below. The dual-branch color point cloud 3D texture mesh reconstruction device described below and the dual-branch color point cloud 3D texture mesh reconstruction method described above can be referred to in correspondence with each other.

[0136] like Figure 7 As shown, the dual-branch color point cloud 3D texture mesh reconstruction device provided in this embodiment of the invention specifically includes:

[0137] The point cloud processing module 310 is used to perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and textureless 3D meshes.

[0138] The consistency repair module 320 is used to perform multi-view consistency repair on multi-view sparse images and generate an initial UV texture map.

[0139] The texture repair module 330 is used to repair the initial UV texture map based on the back projection and weighted blending algorithm to generate a complete UV texture map.

[0140] Enhancement module 340 is used to perform texture resolution enhancement processing on the complete UV texture map to obtain the final UV texture map.

[0141] The generation module 350 is used to apply the final UV texture map back to the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0142] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated further here.

[0143] like Figure 8 As shown, the electronic device provided in this embodiment of the invention may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a dual-branch color point cloud 3D texture mesh reconstruction method, including: performing multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh; performing multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map; performing texture repair on the initial UV texture map based on back projection and weighted mixing algorithms to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and applying the final UV texture map back onto the surface of the textureless 3D mesh to generate a 3D texture mesh.

[0144] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute a dual-branch color point cloud 3D texture mesh reconstruction method, including: performing multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh; performing multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map; performing texture repair on the initial UV texture map based on a back-projection and weighted mixing algorithm to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and applying the final UV texture map back onto the surface of the textureless 3D mesh to generate a 3D texture mesh.

[0146] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for reconstructing a dual-branch color point cloud 3D textured mesh, comprising: performing multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh; performing multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map; performing texture repair on the initial UV texture map based on a back-projection and weighted mixing algorithm to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and applying the final UV texture map back onto the surface of the textureless 3D mesh to generate a 3D textured mesh.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-branch color point cloud 3D texture mesh reconstruction method, characterized in that, The method comprises the following steps: performing multi-view projection and mesh reconstruction on the input color point cloud respectively to obtain multi-view sparse images and a non-texture 3D mesh; performing multi-view consistency repair on the multi-view sparse images to generate an initial UV texture map, comprising: constructing a multi-view consistency-based 2D diffusion model; inputting the multi-view sparse images and preset key calculation parameters into the multi-view consistency-based 2D diffusion model to generate multiple view-consistent target images through reverse diffusion in the multi-view consistency-based 2D diffusion model; and fusing multi-view information in the multiple view-consistent target images to a texture space to obtain the initial UV texture map; performing texture repair on the initial UV texture map based on a back projection and weighted blending algorithm to generate a complete UV texture map, comprising: rendering a plurality of position atlas and normal atlas under multiple views from the non-texture 3D mesh; generating a position texture map and a normal texture map through the back projection and weighted blending algorithm according to the position atlas and the normal atlas; and inputting the initial UV texture map, the position texture map and the normal texture map into a pre-constructed texture repair model to obtain the complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map, comprising: dividing the complete UV texture map into a plurality of overlapping patches; performing resolution enhancement and denoising processing on each overlapping patch respectively to obtain a plurality of optimized patches; performing Gaussian weighted fusion on overlapping areas in the plurality of optimized patches to obtain a fusion result; and performing global color correction and detail sharpening post-processing on the fusion result to obtain the final UV texture map; attaching the final UV texture map to the surface of the non-texture 3D mesh to generate a 3D texture mesh.

2. The dual-branch color point cloud 3D texture mesh reconstruction method of claim 1, wherein, performing multi-view projection on the input color point cloud to obtain multi-view sparse images, comprising: for each view, based on preset camera parameters, transforming the color point cloud to the camera coordinate system and extracting visible points on the convex hull to obtain a visible point cloud subset under each view; projecting the visible points in the visible point cloud subset to a two-dimensional image plane, calculating the pixel coordinates corresponding to each visible point, and filling the color according to the color value of each visible point at the corresponding pixel coordinate position to generate a sparse image under each view; all sparse images under all views are taken as multi-view sparse images.

3. The dual-branch color point cloud 3D texture mesh reconstruction method of claim 1, wherein, generating multiple view-consistent target images through reverse diffusion in the multi-view consistency-based 2D diffusion model, comprising: based on the multi-view sparse images and preset key calculation parameters, initializing a noise image for each view, and defining a degradation path from an original image to a noise image through a forward diffusion process to establish a multi-view joint probability distribution; starting from a full noise state, gradually performing reverse diffusion according to the multi-view joint probability distribution, gradually recovering image details as the number of reverse diffusion steps decreases, while continuously strengthening the multi-view consistency constraint, and finally outputting multiple view-consistent target images.

4. The dual-branch color point cloud 3D texture mesh reconstruction method of claim 1, wherein, Fusing multi-view information in the plurality of target images with consistent views into a texture space to obtain an initial UV texture map, comprising: performing a back projection operation on each target image to obtain a back projection result under each view; mixing the back projection result under each view into a single UV map through a weighted average algorithm of incident angles to obtain the initial UV texture map.

5. The dual-branch color point cloud 3D texture mesh reconstruction method of claim 1, wherein, performing mesh reconstruction on the input color point cloud to obtain a textureless 3D mesh, comprising: inputting the color point cloud into a pre-constructed POPO surface reconstruction model, extracting geometric features and separating color information in the color point cloud through the POPO surface reconstruction model, predicting surface normals using point-based representation and object prior knowledge, and obtaining a textureless 3D mesh after surface interpolation, meshing and optimization processing.

6. A dual-branch color point cloud 3D texture mesh reconstruction apparatus, characterized in that, comprising: a point cloud processing module for performing multi-view projection and mesh reconstruction on the input color point cloud respectively to obtain multi-view sparse images and a textureless 3D mesh; a consistency repair module for repairing multi-view consistency on the multi-view sparse images to generate an initial UV texture map, comprising: constructing a 2D diffusion model based on multi-view consistency; inputting the multi-view sparse images and preset key calculation parameters into the 2D diffusion model based on multi-view consistency to generate a plurality of target images with consistent views through reverse diffusion in the 2D diffusion model based on multi-view consistency; and fusing multi-view information in the plurality of target images with consistent views into a texture space to obtain an initial UV texture map; a texture repair module for repairing texture based on back projection and weighted mixing algorithm on the initial UV texture map to generate a complete UV texture map, comprising: rendering a plurality of position atlas and normal atlas under different views from the textureless 3D mesh; generating position texture map and normal texture map through back projection and weighted mixing algorithm according to the position atlas and normal atlas; inputting the initial UV texture map, the position texture map and the normal texture map into a pre-constructed texture repair model to obtain a complete UV texture map; an enhancement module for performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map, comprising: dividing the complete UV texture map into a plurality of overlapping patches; performing resolution enhancement and denoising processing on each overlapping patch respectively to obtain a plurality of optimized patches; performing Gaussian weighted fusion on overlapping areas in the plurality of optimized patches to obtain a fusion result; and performing global color correction and detail sharpening post-processing on the fusion result to obtain a final UV texture map; a generation module for pasting the final UV texture map back to the surface of the textureless 3D mesh to generate a 3D texture mesh.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the double-branch color point cloud 3D texture mesh reconstruction method of any one of claims 1 to 5. The processor executes the program to realize the double-branch color point cloud 3D texture mesh reconstruction method of any one of claims 1 to 5.

Images