Point cloud texture optimization method based on staged optimization and multi-scale iteration enhancement

Abstract

The invention is suitable for the field of computer vision, and provides a point cloud texture optimization method based on staged optimization and multi-scale iteration enhancement, and the method carries out the similarity calculation of an image subjected to diffusion denoising processing and a reference image, thereby enabling a rendered image to be better aligned with the reference image. Through the process, the rendering quality of the 3DGS is continuously improved, and the content concept in the reference image is better learned.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a method for enhancing point cloud texture optimization based on staged optimization and multi-scale iterative enhancement. Background Technology

[0002] SDS loss essentially measures the similarity between a 3DGS rendered image and a given text prompt. The point cloud editing framework based on 3D Gaussian splashing, by combining a finely tuned multimodal pre-trained model with SDS loss, can guide the point cloud editing results to be as close as possible to the reference image.

[0003] However, the markings for specific text prompts during fine-tuning cannot cover all the details of the reference image, resulting in deviations in certain perspectives and detail representations. Secondly, 3DGS textures optimized using SDS loss often exhibit blurring due to over-smoothing. Summary of the Invention

[0004] The purpose of this invention is to provide a point cloud texture optimization method based on phased optimization and multi-scale iterative enhancement, which aims to solve the technical problems existing in the prior art as identified in the background art.

[0005] This invention is implemented as follows: a point cloud texture optimization method based on staged optimization and multi-scale iterative enhancement, the technical solution of which is as follows:

[0006] An initial image is generated by rendering the point cloud using camera poses obtained through random sampling and specific text prompts.

[0007] The generated image is encoded into a noisy latent variable using a CLIP encoder, and then denoised using a two-dimensional diffusion model to restore a clean image. The similarity between the diffusion-denoised image and a reference image is calculated to obtain the CLIP loss based on the diffusion model. This loss function is not optimized simultaneously with the global SDS loss and local SDS loss; instead, the training process is divided into three equal stages, each using a different combination of loss functions to achieve staged optimization.

[0008] In the phased optimization strategy, the first phase uses CLIP loss and local SDS loss at small time steps to ensure the visual effect of the editing result; the second phase switches to CLIP loss and global SDS loss at intermediate time steps to enable the model to learn the detailed information and concepts in the reference image, further strengthening the alignment with the reference image; the third phase returns to the combination of CLIP loss and local SDS loss at larger time steps to ensure that the quality of the generated edited image is stable and consistent with the reference image.

[0009] For each point in the point cloud, a multidimensional descriptor is defined, where the first three dimensions are initialized to the point's RGB color, and the remaining dimensions are used to store other descriptive information. Under camera pose, the point cloud is rendered multiple times using a depth buffer and a rasterizer renderer to obtain multiple feature maps of different sizes. These feature maps are then integrated using the U-Net rendering network to generate the final rendered image.

[0010] The generated final image is input into the finely tuned multimodal pre-trained model, and global SDS loss is applied to constrain the new perspective rendering to ensure that the rendering result is consistent with the reference image in terms of global features.

[0011] Texture enhancement is achieved by adding a regularization term to the objective function to continuously iterate and optimize the point cloud within a selected region. This regularization term penalizes large differences between the optimized texture and the initial texture in the front view, ensuring a natural transition and consistency in the texture.

[0012] The beneficial effects of this invention are:

[0013] By designing and applying CLIP loss based on a diffusion model, the texture details of the edited region are gradually adjusted and optimized in stages, making the edited texture closer to the reference image. In addition, based on a multi-scale iterative enhancement point cloud texture rendering strategy, by combining a differentiable point rasterizer and a U-Net renderer, as well as introducing regularization terms, the point cloud editing results are refined, improving texture detail and quality, and presenting a realistic appearance. Attached Figure Description

[0014] Figure 1 This is a flowchart of a point cloud texture optimization method based on phased optimization and multi-scale iterative enhancement according to the present invention; Figure 2 This is a U-Net rendered mesh structure diagram. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] like Figure 1 and Figure 2 As shown, a point cloud texture optimization method based on staged optimization and multi-scale iterative enhancement specifically includes:

[0017] Step S1: Using randomly sampled camera pose p and specific text prompts Render the point cloud P to generate an image. ;

[0018] In this embodiment, the camera pose p, including the camera's position and orientation, is first obtained through random sampling. Sampling of the camera pose p can be achieved by uniform sampling in the camera pose space. Then, the point cloud P is rendered using specific text prompts to generate an initial image.

[0019] Step S2: Use the CLIP encoder to process the image Encoding as noisy latent variables And through a two-dimensional diffusion model, the noisy latent variables are... Denoising restores a clean image Calculate clean image Front view with reference image set Similarity;

[0020] Specifically, the CLIP encoder encodes an image into a feature vector containing semantic information about the image. The two-dimensional diffusion model is a U-Net-based neural network used to progressively remove noise and restore a clean image.

[0021] The similarity between the diffusion-denoised image and the reference image is calculated, and the loss function is defined as follows:

[0022] ;

[0023] in, It is a CLIP loss based on a diffusion model. This indicates the CLIP image encoder. It is the generated image after denoising using a diffusion model. It is the front view of the reference image set.

[0024] In this embodiment, CLIP loss Global SDS loss and local SDS loss The optimization is not simultaneous. The training process is divided into three phases on average, each using a different combination of loss functions:

[0025] Phase 1: At small time steps, use global SDS loss and local SDS loss To ensure the visual effect of the editing results.

[0026] Phase 2: At the intermediate time step, switch to CLIP loss. This allows the model to learn detailed information and concepts from the reference image, further enhancing its alignment with the reference image.

[0027] Phase 3: At a larger time step, return to the global SDS loss. and local SDS loss The combination of these elements ensures that the quality of the generated edited image is stable and consistent with the reference image.

[0028] Step S3: Define a multidimensional descriptor for each point cloud P. Under the camera pose, render the point cloud P multiple times using a depth buffer and rasterization renderer to obtain multiple feature maps of different sizes. ;

[0029] For each point in the point cloud P, a 19-dimensional descriptor D is defined, where the first three dimensions are initialized to the point's RGB color. The remaining 16 dimensions are used to store other descriptive information. The descriptor serves as the pseudo-color of the image. Under camera pose p, the point cloud is rendered multiple times using a depth buffer (z-buffer) and a rasterizer, resulting in feature maps of several different sizes.

[0030] ;

[0031] in, and These represent the width and height of the image, respectively. The number of times the point cloud is rendered using rasterization.

[0032] Step S4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full The feature maps are concatenated, and then rendered into the final image using the U-Net rendering network. ;

[0033] The obtained K feature maps are concatenated by channel to obtain a feature tensor. The concatenated feature maps are then rendered into the final image using the U-Net rendering network. :

[0034] ;

[0035] ;

[0036] in, It is a differentiable point cloud rasterizer used to generate feature maps at different resolutions. It is the dimension of the descriptor.

[0037] The U-Net rendering mesh uses gated convolutions, which include three downsampling and upsampling operations to integrate feature maps from different scales and ultimately output an RGB image.

[0038] Step S5: Final image Input the finely tuned multimodal pre-trained model, apply a global loss function, and constrain the rendering of the new perspective;

[0039] The final image generated by S4 The finely tuned text is input into the image T2I model, and a global SDS loss is applied. The specific loss function is as follows:

[0040] .

[0041] Step S6: The objective function of the multimodal pre-trained model Regularization is added to select point clouds within a region to accept gradients, thus affecting the objective function of the multimodal pre-trained model. Perform texture enhancement.

[0042] objective function An additional regularization term is added to penalize large differences between the optimized texture and the initial texture in the front view, in order to ensure a natural transition and consistency of the texture:

[0043] ;

[0044] ;

[0045] in, It is the front view of the reference image set. It is a texture image generated by rendering.

[0046] Selected area The point cloud within the layer receives gradients and performs continuous iterative optimization for texture enhancement, as shown in the equation:

[0047] .

[0048] The above embodiments describe in detail the specific implementation steps of the present invention. Through the above methods, the problems of noise and instability that easily occur when texture changes in point cloud editing can be effectively solved, achieving a more delicate and coherent texture optimization effect and improving the visual quality of point cloud editing results.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0050] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for enhancing point cloud texture optimization based on staged optimization and multi-scale iterative enhancement, characterized in that, The method includes: Step S1: Camera pose obtained through random sampling and specific text prompts Render the point cloud P to generate an image. ; Step S2: Use a CLIP encoder to process the image. Encoding as noisy latent variables And through a two-dimensional diffusion model, the noisy latent variables are... Denoising restores a clean image Calculate clean image Front view with reference image set Similarity; Step S3: Define a multidimensional descriptor for each point cloud P. Under camera pose, render the point cloud P using a depth buffer and rasterization renderer to obtain several feature maps of different sizes. ; Step S4: Transfer the feature map The network is stitched together and rendered using U-Net. The stitched feature maps are then rendered into the final image. ; Step S5: Final image Input a multimodal pre-trained model, apply a global loss function, and constrain the rendering of new perspectives; Step S6: In the objective function of the multimodal pre-trained model Regularization is added to select the point cloud within the region to receive gradients, thereby enhancing the texture of the objective function of the multimodal pre-trained model.

2. The method according to claim 1, characterized in that, The calculation of the clean image in step S2 Front view with reference image set The similarity is further used to align the edited result with the parametric image. The loss function is defined as follows: ； in, It is a CLIP loss based on a diffusion model. This indicates the CLIP image encoder. It is the generated image after denoising using a diffusion model. It is the front view of the reference image set.

3. The method according to claim 2, characterized in that, During the process of aligning the edited result with the parametric image, CLIP loss Global SDS loss and local SDS loss The training process is divided into three phases on average, with each phase using a different combination of loss functions: Phase 1: At small time steps, use global SDS loss and local SDS loss To ensure the visual quality of the edited results; Phase 2: At the intermediate time step, switch to CLIP loss. This enables the model to learn detailed information and concepts from the reference image, thus strengthening its alignment with the reference image; Phase 3: At a large time step, return to the global SDS loss. and local SDS loss The combination of these elements ensures that the generated edited image has stable quality and remains consistent with the reference image.

4. The method according to claim 1, characterized in that, Step S3 includes: Define a 19-dimensional descriptor D for each point in the point cloud P, where the first three dimensions are initialized to the RGB color of the point; The descriptor serves as the pseudo-color of the image. Under camera pose p, the point cloud is rendered multiple times using a depth buffer and a rasterization renderer to obtain several feature maps of different sizes. ; ； in, and These represent the width and height of the image, respectively, and K is the number of times the point cloud is rasterized for rendering.

5. The method according to claim 1, characterized in that, In step S4, the rendering network via U-Net... The stitched feature maps are then rendered into the final image. : ； ； in, It is a differentiable point cloud rasterizer used to generate feature maps at different resolutions, where D is the dimension of the descriptor. Indicates camera pose parameters Below, from the point cloud parameter set The final image generated by differentiable rendering; U-Net rendering mesh Gated convolutions are employed, comprising three downsampling and upsampling operations, to integrate feature maps from different scales and output the final image. .

6. The method according to claim 1, characterized in that, The final image Input a multimodal pre-trained model and apply global SDS loss. The loss function is: ； in, This represents the SDS loss function relative to the point cloud parameter set. The gradient; A personalized text-to-image generation model; It is random Gaussian noise, used for denoising prediction during the diffusion process; This refers to the time step in the diffusion model; This is the time step weighting function in the SDS loss; In time step Noisy latent variables; This is a text prompt message containing special markers used to describe the target editing object.

7. The method according to claim 6, characterized in that, The addition of regularization to the objective function of the multimodal pre-trained model penalizes the difference between the optimized texture and the initial texture in the front view, ensuring natural transitions and consistency of the texture. ； ； For the global SDS loss of the texture, This is the texture regularization loss; Select the subset of point cloud parameters corresponding to the target region Accepting gradients, perform continuous iterative optimization for texture enhancement, specifically: ； in, This represents the objective function relative to the point cloud parameter set. The gradient operator.

Images