Double-branch color point cloud 3D texture grid reconstruction method and device and electronic equipment

Abstract

The invention belongs to the technical field of image processing, and provides a double-branch color point cloud 3D texture grid reconstruction method and device and electronic equipment, and the method comprises the steps: carrying out the multi-view projection and grid reconstruction of an input color point cloud, and obtaining a multi-view sparse image and a texture-free 3D grid; performing multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map; based on a back projection and weighted hybrid algorithm, performing texture repair on the initial UV texture map to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and pasting the final UV texture map back to the surface of the texture-free 3D grid to generate a 3D texture grid. According to the scheme, through collaborative optimization of the double-branch frame in the image space and the texture space, the texture definition and the global consistency are improved by utilizing multi-view consistency repair, and the definition of the texture map is improved through texture repair and texture resolution enhancement.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dual-branch color point cloud 3D texture mesh reconstruction method, device and electronic equipment. Background Art

[0002] In recent years, with the rapid development of technologies such as virtual reality, augmented reality, digital twins, and the metaverse, 3D textured mesh reconstruction based on color point clouds has attracted widespread attention in the fields of computer vision and graphics. Color point clouds, as a 3D data representation containing geometric coordinates and color information, can be converted into textured 3D mesh models through appropriate processing. These point clouds are widely used in scenarios such as cultural heritage preservation, 3D modeling, and real-time rendering.

[0003] In the related art, one type of method achieves texture reconstruction by learning implicit representations of color information in three-dimensional space or UV space. This type of method usually requires a large amount of three-dimensional data, and the acquisition cost of three-dimensional data is high and the quantity is limited, resulting in poor generalization ability of the model when dealing with unseen objects or complex scenes. In addition, because the implicit representation is difficult to accurately capture complex geometric details, the reconstruction results often have texture blurring problems, especially at occlusion boundaries or high-frequency texture areas. Another type of method uses traditional surface reconstruction technology, such as Poisson surface reconstruction, to directly generate texture meshes by fusing point cloud color information. However, this method relies on accurate point normal estimation, and is prone to geometric redundancy or color distortion when dealing with sparse point clouds or complex topologies, making it difficult to meet the needs of high-quality texture mesh construction.

[0004] At the same time, the field of two-dimensional vision has made significant progress in image super-resolution and generation technology. In particular, the emergence of diffusion models has provided strong prior support for image restoration and generation. Compared with three-dimensional data, two-dimensional image data is abundant and easy to obtain, and the technology is more mature. However, the introduction of two-dimensional diffusion models into three-dimensional generation tasks still mainly predicts or optimizes color information in three-dimensional space, which is time-consuming and difficult to eliminate texture blurring. In addition, existing diffusion models mostly use single-view images as input and lack consideration of other perspectives when processing multi-view images. This leads to insufficient global consistency and often causes problems such as occlusion boundary artifacts, boundary texture misalignment, "seams" and repeated multi-faceted features.

[0005] Therefore, the traditional scheme of reconstructing texture meshes from color point clouds has technical problems such as low texture clarity, poor global consistency and insufficient generalization ability. Summary of the Invention

[0006] The present invention provides a dual-branch color point cloud 3D texture mesh reconstruction method, device and electronic device to solve the defects of traditional solutions for reconstructing texture meshes from color point clouds, such as low texture clarity, poor global consistency and insufficient generalization ability.

[0007] In one aspect, the present invention provides a dual-branch color point cloud 3D texture mesh reconstruction method, comprising: Perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and texture-free 3D meshes; Performing multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map; Performing texture repair on the initial UV texture map based on back-projection and weighted blending algorithms to generate a complete UV texture map; Performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; The final UV texture map is pasted back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0008] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multi-view projection is performed on the input color point cloud to obtain a multi-view sparse image, including: For each viewing angle, based on the preset camera parameters, the colored point cloud is transformed into the camera coordinate system, and the visible points on the convex hull are extracted to obtain the visible point cloud subset at each viewing angle; Projecting the visible points in the visible point cloud subset onto a two-dimensional image plane, calculating the pixel coordinates corresponding to each visible point, and filling the corresponding pixel coordinate position with color according to the color value of each visible point to generate a sparse image at each viewing angle; The sparse images under all views are regarded as multi-view sparse images.

[0009] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multi-view consistency restoration is performed on the multi-view sparse image to generate an initial UV texture map, including: Construct a 2D diffusion model based on multi-view consistency; Inputting the multi-view sparse image and preset key calculation parameters into the multi-view consistency-based 2D diffusion model, so as to generate a plurality of target images with consistent viewpoints through reverse diffusion in the multi-view consistency-based 2D diffusion model; The multi-view information in the multiple target images with the same viewpoint is fused into the texture space to obtain an initial UV texture map.

[0010] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multiple target images with consistent perspectives are generated by reverse diffusion in the 2D diffusion model based on multi-perspective consistency, including: Based on the multi-view sparse images and preset key calculation parameters, a noise image is initialized for each view, and a degradation path from the original image to the noise image is defined through a forward diffusion process to establish a multi-view joint probability distribution; Starting from a fully noisy state, reverse diffusion is performed step by step according to the multi-view joint probability distribution. As the number of reverse diffusion steps decreases, image details are gradually restored while continuously strengthening the multi-view consistency constraints, ultimately outputting multiple target images with consistent viewpoints.

[0011] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, multi-view information in the multiple target images with consistent viewpoints is fused into the texture space to obtain an initial UV texture map, including: Perform a back-projection operation on the target image at each viewing angle to obtain a back-projection result at each viewing angle; The back-projection results at each viewing angle are mixed into a single UV mapping through a weighted average algorithm of the incident angle to obtain the initial UV texture map.

[0012] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, the initial UV texture map is texture repaired based on back projection and weighted hybrid algorithm to generate a complete UV texture map, including: Rendering the untextured 3D mesh to obtain position atlases and normal atlases under multiple viewing angles; Generating a position texture map and a normal texture map by back-projection and weighted blending algorithm according to the position atlas and the normal atlas; The initial UV texture map, the position texture map, and the normal texture map are all input into a pre-built texture repair model to obtain a complete UV texture map.

[0013] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, texture resolution enhancement processing is performed on the complete UV texture map to obtain a final UV texture map, including: Splitting the complete UV texture map into a plurality of overlapping patches; Perform resolution enhancement and denoising on each overlapping patch separately to obtain multiple optimized patches; Performing Gaussian weighted fusion on overlapping areas of the multiple optimized patches to obtain a fusion result; The fusion result is subjected to global color correction and detail sharpening post-processing to obtain a final UV texture map.

[0014] According to the dual-branch color point cloud 3D texture mesh reconstruction method provided by the present invention, mesh reconstruction is performed on the input color point cloud to obtain a texture-free 3D mesh, including: The color point cloud is input into a pre-built POPO surface reconstruction model, and the geometric features in the color point cloud are extracted and the color information is separated by the POPO surface reconstruction model. The surface normal is predicted using point-based representation and object prior knowledge, and a textureless 3D mesh is obtained after surface interpolation, meshing and optimization.

[0015] On the other hand, the present invention also provides a dual-branch color point cloud 3D texture mesh reconstruction device, comprising: The point cloud processing module is used to perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and texture-free 3D meshes; A consistency restoration module, configured to perform multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map; A texture restoration module, configured to perform texture restoration on the initial UV texture map based on a back-projection and weighted blending algorithm to generate a complete UV texture map; An enhancement module, configured to perform texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; A generation module is used to paste the final UV texture map back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0016] On the other hand, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, it implements any of the above-described dual-branch color point cloud 3D texture mesh reconstruction methods.

[0017] The dual-branch color point cloud 3D texture mesh reconstruction method, device, and electronic device provided by the present invention perform multi-perspective projection and mesh reconstruction on the input color point cloud to obtain a multi-perspective sparse image and a texture-free 3D mesh. The multi-perspective sparse image is then subjected to multi-perspective consistency restoration to generate an initial UV texture map. The initial UV texture map is texture-restored based on a back-projection and weighted hybrid algorithm to generate a complete UV texture map. The complete UV texture map is texture-resolution enhanced to obtain a final UV texture map. The final UV texture map is then applied back to the surface of the texture-free 3D mesh to generate a 3D texture mesh. This solution utilizes a dual-branch framework to collaboratively optimize image and texture space. It utilizes multi-perspective consistency restoration to improve texture clarity and global consistency. Furthermore, through texture restoration and texture resolution enhancement, the clarity of the texture map is enhanced, making it adaptable to different task requirements. This effectively addresses the problems of low texture clarity, poor global consistency, and insufficient generalization capabilities in traditional texture mesh reconstruction solutions from color point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 3D texture mesh reconstruction method of a dual-branch color point cloud provided by an embodiment of the present invention; Figure 2 3D texture mesh reconstruction method of a dual-branch color point cloud provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the implementation principle of the 2D diffusion model based on multi-view consistency; Figure 4 This is a comparison chart of the results of multi-view consistency optimization in image space; Figure 5 This is a comparison chart of the results of texture space optimization and resolution enhancement; Figure 6 is a schematic diagram of the visualization results of the comparison between the method provided in this embodiment and the traditional method; Figure 7 3D texture mesh reconstruction device with dual-branch color point cloud provided by an embodiment of the present invention; Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following combination Figures 1 to 8 The detailed scheme of the dual-branch color point cloud 3D texture mesh reconstruction method, device and electronic device provided by the embodiments of the present invention is described.

[0022] like Figure 1 As shown, the dual-branch color point cloud 3D texture mesh reconstruction method provided by the embodiment of the present invention mainly includes the following steps: Step 110: Perform multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a texture-free 3D mesh.

[0023] Step 120: Perform multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map.

[0024] It should be noted that UV mainly refers to the UV coordinate system. In the UV coordinate system, the texture image can be regarded as a 2D plane, U corresponds to the horizontal direction, similar to the X-axis of the plane rectangular coordinate system, and V corresponds to the vertical direction, similar to the Y-axis of the plane rectangular coordinate system.

[0025] Step 130: Based on the back-projection and weighted blending algorithm, the initial UV texture map is texture repaired to generate a complete UV texture map.

[0026] Step 140: Perform texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map.

[0027] Step 150: Map the final UV texture map back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0028] The dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment can be found in the specific implementation framework. Figure 2The reconstruction method can be executed in the image space module 210 and the texture space module 220, respectively. The image space module 210 includes a multi-view projection and consistency restoration stage, and the texture space module 220 includes a texture restoration and texture resolution enhancement stage. In the image space module 210, an initial UV texture map can be generated using an image restoration diffusion model (i.e., a 2D diffusion model with multi-view consistency). In the texture space module 220, texture details can be optimized using a texture restoration diffusion model (i.e., a texture restoration model), and the resolution can be optionally enhanced to generate a final texture map. Finally, the final texture map is pasted back onto the untextured mesh to generate a high-quality 3D textured mesh.

[0029] In one embodiment, performing multi-view projection on the input color point cloud to obtain a multi-view sparse image includes: First, for each viewpoint, the color point cloud is transformed into the camera coordinate system based on the preset camera parameters, and the visible points on the convex hull are extracted to obtain the visible point cloud subset at each viewpoint.

[0030] Then, the visible points in the visible point cloud subset are projected onto the two-dimensional image plane, the pixel coordinates corresponding to each visible point are calculated, and the color is filled in the corresponding pixel coordinate position according to the color value of each visible point to generate a sparse image at each viewing angle.

[0031] Finally, the sparse images from all views are taken as multi-view sparse images.

[0032] This embodiment can input color point cloud S Perform multi-view projection to generate M Sparse images under different view angles, where M is the preset number of viewing angles. The color point cloud contains the three-dimensional coordinates and RGB color values ​​of each point. First, in order to avoid the projection containing occluded points, this embodiment adopts a hidden point removal operation. Specifically, for each viewing angle, based on its camera parameters, the color point cloud is transformed into the camera coordinate system, and the visible points on the convex hull are extracted to obtain the subset of visible point clouds under this viewing angle. Subsequently, the visible points are projected onto the two-dimensional image plane, the pixel coordinates corresponding to each visible point are calculated, and the color is filled in at the corresponding pixel coordinate position according to the color value of the visible point to generate a sparse image under this viewing angle. Repeating this process can generate M Sparse image In actual applications, the viewing angle adopts the Fibonacci sphere distribution to ensure uniform coverage.

[0033] In one embodiment, performing multi-view consistency restoration on a multi-view sparse image to generate an initial UV texture map specifically includes: The first step is to construct a 2D diffusion model based on multi-view consistency.

[0034] It can be understood that this is a novel zero-shot framework for any image restoration problem, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring, compared to the existing denoising diffusion null space model. Specifically, the denoising diffusion null space model is implemented based on the denoising diffusion probability model. It gradually adds noise to the data through a forward process and reconstructs the target image from the noise in a backward process. It utilizes range-null space decomposition to ensure data consistency, while it iteratively optimizing the null space content to generate texture details that conform to the natural image distribution.

[0035] On this basis, the present invention analyzes the impact of multi-perspective consistency on texture reconstruction for sparse images generated by multi-perspective projection of color point clouds. For example, inconsistent colors and boundaries between different perspectives can lead to blurred textures or seam artifacts, while insufficient number of perspectives may result in loss of global details. Combined with the multi-perspective consistency constraint, this embodiment proposes a 2D diffusion model based on multi-perspective consistency for texture generation. By jointly optimizing the multi-perspective sparse images obtained by projection, it ensures that the generated initial UV texture map achieves relatively good results in both global consistency and detail clarity.

[0036] In the second step, the multi-view sparse images and preset key calculation parameters are input into the 2D diffusion model based on multi-view consistency to generate multiple target images with consistent viewpoints through reverse diffusion in the 2D diffusion model based on multi-view consistency.

[0037] In a specific implementation, generating multiple target images with consistent perspectives through reverse diffusion in a 2D diffusion model based on multi-perspective consistency includes: Firstly, based on multi-view sparse images and preset key calculation parameters, a noisy image is initialized for each view, and the degradation path from the original image to the noisy image is defined through the forward diffusion process to establish a multi-view joint probability distribution.

[0038] Then, starting from the full noise state, reverse diffusion is performed step by step according to the multi-view joint probability distribution. As the number of reverse diffusion steps decreases, the image details are gradually restored, while the multi-view consistency constraints are continuously strengthened, and finally multiple target images with consistent perspectives are output.

[0039] In practical applications, such as Figure 3 As shown, we first need to define the model input and initialize it. The model input mainly includes M Sparse image And the corresponding key calculation parameters, specifically, the key calculation parameters include: camera pose and degradation factor A m , where m=1,2,...,M. At the same time, initialize the noise images under M perspectives , T is the number of steps in the entire diffusion process, and t represents each step of the reverse diffusion.

[0040] After that, the joint probability distribution of images under M perspectives is modeled , generates M consistent dense images through a single back-diffusion. The forward process adds noise to each view independently, defining the degradation path from sparse image to noisy image. The specific formula is as follows: (1) (2) in, is the probability of multi-view joint forward diffusion, is a set of original images from M perspectives, is the set of noise images from M perspectives after diffusion from step 1 to step T, is the probability distribution of diffusion at step t from a single perspective, is the noise scheduling parameter at time step t, is the noise image of the m-th view at time step t. The reverse diffusion process is defined as: (3) (4) in, is the joint probability of reverse diffusion, is the complete sequence of M viewpoints from the original image to the final noisy image, is the initial noise distribution, is the mean of the mth perspective, depending on the intermediate state of all perspectives , the calculation formula is: (5) in, , , is the noise scheduling parameter at time step t, is the noise predictor for the m-th view.

[0041] Then, in each step of reverse diffusion, to ensure multi-view Figure 1 To ensure consistency, the model uses 3D-aware feature attention mechanism to synchronize the features of M views. First, from the intermediate state of M views Extract features and build a The spatial feature stereo microscope is a convex stereo microscope, where V is the resolution of the spatial feature stereo microscope. Then, for each view angle m, a cone stereo microscope is constructed according to its camera pose, and features are extracted from the spatial feature stereo microscope by interpolation. On the intermediate feature map of the image, the depth-direction attention mechanism is applied to extract features along the depth dimension of the cone to ensure multi-view Figure 1 Consistency.

[0042] In addition, to further ensure the data consistency of the generated image, the model adopts range-null space decomposition. Specifically, for the noise predictor Estimated dense image Apply range-space decomposition to perform data consistency correction. The specific formula is as follows: (6) in, is the degenerate operator The pseudo-rebellion, Ensure that the range space meets data consistency represents the null space. Then, the rectified Convert back to the noise state for the next step of denoising.

[0043] After T steps of reverse diffusion, the model generates M target images with consistent multi-viewpoints. ,These images not only meet data consistency but also maintain geometric and color consistency in 3D space, and can be directly used for subsequent 3D reconstruction or other multi-view applications. Figure 4 The visualization results are with (i.e., part a) and without (i.e., part b) multi-view consistency.

[0044] In the third step, the multi-view information in multiple target images with the same viewpoint is fused into the texture space to obtain the initial UV texture map.

[0045] After obtaining the target image with consistent multi-viewpoints, it is necessary to further fuse the multi-viewpoint information in the target image into the texture space to generate the initial UV texture map. , the specific process includes back-projection and weighted blending operations based on the incident angle.

[0046] In a specific implementation, the multi-view information of multiple target images with the same viewpoint is fused into the texture space to obtain the initial UV texture map, which specifically includes: First, a back-projection operation is performed on the target image at each viewing angle to obtain a back-projection result at each viewing angle.

[0047] In this embodiment, for each non-background pixel in the target image, its corresponding grid surface point is determined, and its value is mapped to the corresponding coordinate in the UV space. The specific formula is as follows: (7) in, Represents the pixel coordinates in the 2D image, K represents the intrinsic parameter matrix, R and T represent the camera extrinsic parameter matrix, Representing rays and meshes The intersection calculation, represents the back-projection operation, Represents the coordinates in UV space.

[0048] Then, the back-projection results at each view angle are mixed into a single UV mapping through a weighted average algorithm of the incident angle to obtain the initial UV texture map.

[0049] This embodiment uses a weighted average of the incident angles to blend the back-projections into a single UV map to optimize the fusion quality. Specifically, the cosine similarity between the viewing direction and the pixel-by-pixel normal vector in image space is used to determine the weighted contribution of each pixel to the blended texture. The specific formula for weighted averaging is as follows: (8) in, Represents the color value of pixel p on the fused UV texture map, Indicates that the image of the i-th perspective The pixel p is back-projected onto the UV texture map, Indicates pixel p in the i-th viewing angle image The incident angle in The exponent representing the weighted average is used to control the degree of influence of the incident angle on the weight. Represents a small constant used to avoid the denominator being zero. Specifically, Can be designed as 6, the incident angle is defined as follows: (9) in, is a vector and The angle between is the viewing direction from camera view i to pixel p, is the normal vector at pixel p.

[0050] In one embodiment, mesh reconstruction is performed on the input color point cloud to obtain a texture-free 3D mesh, specifically including: The color point cloud is input into the pre-built POPO surface reconstruction model. The geometric features in the color point cloud are extracted and the color information is separated by the POPO surface reconstruction model. The surface normal is predicted using the point-based representation and object prior knowledge. After surface interpolation, meshing and optimization, a textureless 3D mesh is obtained.

[0051] In this embodiment, the color point cloud SAs input, the color point cloud preprocessed by the POPO (Point-based Object Priors) surface reconstruction model is used to generate a textureless 3D mesh. The model network structure is as follows: (10) in, represents the implicit representation of the surface at the query point, represents the i-th point in the colored point cloud, The nearest neighbor point set represented by , calculates the local geometric relationship based on the learned difference weights, Represents the point cloud convolution operation, the specific operations are as follows: (11) in, for The input features of , Conv is the point cloud convolution operation.

[0052] In one embodiment, texture repair is performed on the initial UV texture map based on a back-projection and weighted blending algorithm to generate a complete UV texture map, including: In the first step, position atlases and normal atlases under multiple view angles are rendered from the untextured 3D mesh.

[0053] In the second step, based on the position atlas and normal atlas, position texture map and normal texture map are generated through back projection and weighted blending algorithm.

[0054] In this embodiment, first, a textureless 3D mesh is Rendering to obtain the position atlas of M perspectives and normal atlas The position atlas is then transformed into and normal atlas Fuse into texture space to generate corresponding position texture map and normal texture maps , which is used to input the texture restoration model together to achieve geometry-guided texture detail restoration.

[0055] In the third step, the initial UV texture map, position texture map, and normal texture map are input into the pre-built texture repair model to obtain a complete UV texture map.

[0056] The initial UV texture map is generated by simply using back projection and weighted blending based on the angle of incidence. , because the selected view in the early stage does not cover the 3D shape structure enough, it may cause occlusion, resulting in missing areas of the texture map; at the same time, when the pixels of the image are back-projected into the UV space, there is not a one-to-one correspondence, which may cause pixel-level "holes" on the texture map. In this embodiment, the initial UV texture map with position texture maps and normal texture maps The texture restoration model is jointly input to achieve geometry-guided texture detail restoration.

[0057] Specifically, the texture restoration model is a potential diffusion model based on U-Net, which is fine-tuned from the Emu model to obtain the texture restoration diffusion network, denoted as ,The structure of the texture restoration model is as follows: (12) in, As part of the input parameters of the texture repair model, z is a 2D noise map where each pixel is independently sampled from a standard Gaussian distribution. Is the output of the model, the complete UV texture map The loss is calculated with the real texture map to optimize the diffusion network.

[0058] This embodiment generates a textureless 3D mesh by mesh reconstruction technology, and optimizes the initial UV texture map by combining the texture restoration model based on back projection and weighted hybrid algorithm to generate a complete UV texture map. The processing results corresponding to the texture space optimization and enhancement steps, as well as the processing results after both are adopted, are as follows: Figure 5 shown.

[0059] In one embodiment, a texture resolution enhancement process is performed on the complete UV texture map to obtain a final UV texture map, specifically including: The first step is to split the full UV texture map into multiple overlapping patches.

[0060] In the second step, resolution enhancement and denoising are performed on each overlapping patch to obtain multiple optimized patches.

[0061] This embodiment enhances the complete UV texture map by a patch-based approach In practical applications, target texture maps with corresponding resolutions can be generated according to different task requirements. Specifically, the complete UV texture map The image is divided into multiple overlapping patches and the Multi Diffusion model is used as a pre-trained diffusion model. The MultiDiffusion model is a multimodal diffusion model library. Its core idea is to use the diffusion process to gradually build complex structures and simulate the interaction between different modes in reality. In this embodiment, the Multi Diffusion model is used to enhance the resolution of each patch and generate a high-resolution output through the denoising process. In this process, the position texture map and normal texture maps As a geometric guide condition.

[0062] In the third step, Gaussian weighted fusion is performed on the overlapping areas of multiple optimized patches to obtain the fusion result.

[0063] In this embodiment, overlapping areas are fused through Gaussian weighting to avoid seams and inconsistencies.

[0064] The fourth step is to perform global color correction and detail sharpening on the fusion result to obtain the final UV texture map.

[0065] In practical applications, color correction can be achieved through histogram matching or global brightness adjustment, maintaining consistency between texture and original point cloud color information. Detail sharpening can enhance texture edges and surface details through high-frequency filtering. The resulting high-resolution final UV texture map preserves the clarity of local details while maintaining global consistency with the 3D object's geometry and semantics, making it suitable for applications such as high-quality rendering and virtual reality.

[0066] This embodiment introduces an optional texture enhancement network to improve the resolution of the complete UV texture map, and maps the final UV texture map to the surface of the untextured mesh to generate a 3D textured mesh.

[0067] In order to verify the processing effect achieved by the dual-branch color point cloud 3D texture mesh reconstruction method provided in this embodiment, this embodiment uses several traditional methods such as SPR, NKSR, Texture Field and the method provided in this embodiment (denoted as Ours) to process the same object samples to be reconstructed (denoted as Ground Truth, i.e., real information) consisting of different types of objects such as packaging boxes, backpacks, toy cars and clocks. The visual comparison results are shown in the figure below. Figure 6 shown.

[0068] See also Figure 6As a traditional 3D reconstruction method, SPR (Single-View Physically-Based Rendering) has obvious defects in the reconstruction results of the above-mentioned object samples to be reconstructed, such as blurred clock surface and lost backpack texture. As a reconstruction method based on neural implicit representation, NKSR (Neural Kernel Surface Reconstruction) has better processing effect than SPR, but there are still missing details, such as the color of the toy car and the texture of the clock. TextureField is a method of focused texture reconstruction. In the corresponding processing results, some objects (such as packaging boxes) have lost their original patterns, and the clock is almost left blank, indicating that the processing of complex textures is insufficient. The method provided in this embodiment (i.e. Figure 6 Ours) reconstruction effect is closer to the real information (i.e. Figure 6 The texture and shape of objects are fully preserved, such as the backpack pattern and clock numbers are clearer.

[0069] In summary, compared to traditional processing methods, the method provided in this embodiment not only enhances mesh texture clarity but also effectively resolves boundary misalignment and seam issues, maintaining the global consistency of 3D objects. Furthermore, it reduces reliance on 3D training data, improves generalization capabilities, and is applicable to a variety of 3D reconstruction scenarios.

[0070] Based on the same general inventive concept, the present invention also protects a dual-branch color point cloud 3D texture mesh reconstruction device. The dual-branch color point cloud 3D texture mesh reconstruction device provided by the present invention is described below. The dual-branch color point cloud 3D texture mesh reconstruction device described below and the dual-branch color point cloud 3D texture mesh reconstruction method described above can be referenced to each other.

[0071] like Figure 7 As shown, the dual-branch color point cloud 3D texture mesh reconstruction device provided by the embodiment of the present invention specifically includes: The point cloud processing module 310 is used to perform multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a texture-free 3D mesh.

[0072] The consistency restoration module 320 is used to perform multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map.

[0073] The texture restoration module 330 is configured to perform texture restoration on the initial UV texture map based on back-projection and weighted blending algorithms to generate a complete UV texture map.

[0074] The enhancement module 340 is used to perform texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map.

[0075] The generation module 350 is used to map the final UV texture map back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

[0076] Regarding the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the relevant methods and will not be elaborated on again here.

[0077] like Figure 8 As shown, the electronic device provided by the embodiment of the present invention may include: a processor (processor) 410, a communication interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logic instructions in the memory 430 to execute a dual-branch color point cloud 3D texture mesh reconstruction method, including: performing multi-view projection and mesh reconstruction on the input color point cloud to obtain a multi-view sparse image and a textureless 3D mesh; performing multi-view consistency repair on the multi-view sparse image to generate an initial UV texture map; performing texture repair on the initial UV texture map based on back projection and weighted mixing algorithms to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and pasting the final UV texture map back onto the surface of the textureless 3D mesh to generate a 3D texture mesh.

[0078] Furthermore, the logic instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a dual-branch color point cloud 3D texture mesh reconstruction method, including: performing multi-perspective projection and mesh reconstruction on the input color point cloud to obtain a multi-perspective sparse image and a texture-free 3D mesh; performing multi-perspective consistency repair on the multi-perspective sparse image to generate an initial UV texture map; based on back projection and weighted mixing algorithms, texture repairing the initial UV texture map to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and pasting the final UV texture map back to the surface of the texture-free 3D mesh to generate a 3D texture mesh.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a dual-branch color point cloud 3D texture mesh reconstruction method, including: performing multi-perspective projection and mesh reconstruction on the input color point cloud to obtain a multi-perspective sparse image and a textureless 3D mesh; performing multi-perspective consistency repair on the multi-perspective sparse image to generate an initial UV texture map; performing texture repair on the initial UV texture map based on back projection and weighted mixing algorithms to generate a complete UV texture map; performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; and pasting the final UV texture map back onto the surface of the textureless 3D mesh to generate a 3D texture mesh.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0082] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A dual-branch color point cloud 3D texture mesh reconstruction method, characterized in that: include: Perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and texture-free 3D meshes; Performing multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map; Performing texture repair on the initial UV texture map based on back-projection and weighted blending algorithms to generate a complete UV texture map; Performing texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; The final UV texture map is pasted back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

2. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 1, characterized in that: Perform multi-view projection on the input color point cloud to obtain a multi-view sparse image, including: For each viewing angle, based on the preset camera parameters, the colored point cloud is transformed into the camera coordinate system, and the visible points on the convex hull are extracted to obtain the visible point cloud subset at each viewing angle; Projecting the visible points in the visible point cloud subset onto a two-dimensional image plane, calculating the pixel coordinates corresponding to each visible point, and filling the corresponding pixel coordinate position with color according to the color value of each visible point to generate a sparse image at each viewing angle; The sparse images under all views are regarded as multi-view sparse images.

3. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 1, characterized in that: Performing multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map, including: Construct a 2D diffusion model based on multi-view consistency; Inputting the multi-view sparse image and preset key calculation parameters into the multi-view consistency-based 2D diffusion model, so as to generate a plurality of target images with consistent viewpoints through reverse diffusion in the multi-view consistency-based 2D diffusion model; The multi-view information in the multiple target images with the same viewpoint is fused into the texture space to obtain an initial UV texture map.

4. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 3, characterized in that: Generating multiple target images with consistent perspectives by reverse diffusion in the multi-perspective consistent 2D diffusion model includes: Based on the multi-view sparse images and preset key calculation parameters, a noise image is initialized for each view, and a degradation path from the original image to the noise image is defined through a forward diffusion process to establish a multi-view joint probability distribution; Starting from a fully noisy state, reverse diffusion is performed step by step according to the multi-view joint probability distribution. As the number of reverse diffusion steps decreases, image details are gradually restored while continuously strengthening the multi-view consistency constraints, ultimately outputting multiple target images with consistent viewpoints.

5. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 3, characterized in that: The multi-view information in the multiple target images with the same viewpoint is fused into the texture space to obtain an initial UV texture map, including: Perform a back-projection operation on the target image at each viewing angle to obtain a back-projection result at each viewing angle; The back-projection results at each viewing angle are mixed into a single UV mapping through a weighted average algorithm of the incident angle to obtain the initial UV texture map.

6. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 1, characterized in that: Based on the back-projection and weighted blending algorithm, the initial UV texture map is texture repaired to generate a complete UV texture map, including: Rendering the untextured 3D mesh to obtain position atlases and normal atlases under multiple viewing angles; Generating a position texture map and a normal texture map by back-projection and weighted blending algorithm according to the position atlas and the normal atlas; The initial UV texture map, the position texture map, and the normal texture map are all input into a pre-built texture repair model to obtain a complete UV texture map.

7. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 1, characterized in that: The complete UV texture map is subjected to texture resolution enhancement processing to obtain a final UV texture map, including: Splitting the complete UV texture map into a plurality of overlapping patches; Perform resolution enhancement and denoising on each overlapping patch separately to obtain multiple optimized patches; Performing Gaussian weighted fusion on overlapping areas of the multiple optimized patches to obtain a fusion result; The fusion result is subjected to global color correction and detail sharpening post-processing to obtain a final UV texture map.

8. The dual-branch color point cloud 3D texture mesh reconstruction method according to claim 1, characterized in that: Reconstruct the mesh of the input color point cloud to obtain a texture-free 3D mesh, including: The color point cloud is input into a pre-built POPO surface reconstruction model, and the geometric features in the color point cloud are extracted and the color information is separated by the POPO surface reconstruction model. The surface normal is predicted using point-based representation and object prior knowledge, and a textureless 3D mesh is obtained after surface interpolation, meshing and optimization.

9. A dual-branch color point cloud 3D texture mesh reconstruction device, characterized in that: include: The point cloud processing module is used to perform multi-view projection and mesh reconstruction on the input color point cloud to obtain multi-view sparse images and texture-free 3D meshes; A consistency restoration module, configured to perform multi-view consistency restoration on the multi-view sparse image to generate an initial UV texture map; A texture restoration module, configured to perform texture restoration on the initial UV texture map based on a back-projection and weighted blending algorithm to generate a complete UV texture map; An enhancement module, configured to perform texture resolution enhancement processing on the complete UV texture map to obtain a final UV texture map; A generation module is used to paste the final UV texture map back onto the surface of the untextured 3D mesh to generate a 3D textured mesh.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the dual-branch color point cloud 3D texture mesh reconstruction method according to any one of claims 1 to 8 is implemented.

Images