A Structure-Aware Multi-View Texture Reconstruction Method and System

Abstract

This invention discloses a structure-aware multi-view texture reshaping method and system, belonging to the field of computer vision processing. The method uses a pre-trained diffusion model combined with a control network to process multi-view rendered images of 3D scenes, encoding image and text descriptions, and extracting structural information to pass to a UNet. A dual-path architecture for reconstruction and editing is constructed, sampling noise for multi-step iterative denoising; a noise sharing module weights and mixes predicted noise in the same batch, and a dual-stream residual guidance module corrects predicted noise in different batches. Finally, the edited image is used to optimize the original 3D scene, achieving consistent texture reshaping across multiple views. This invention fully leverages the powerful generalization ability of the diffusion model, combining structure-aware guidance and a multi-stage consistency modeling mechanism to improve the consistency of texture structure among multiple views during the diffusion generation process. In 3D scene editing tasks, it exhibits highly realistic texture reconstruction capabilities and consistent editing performance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing, and in particular relates to a multi-view texture reshaping method and system based on structure perception. Background Technology

[0002] In fields such as digital content creation, virtual reality, film and television post-production, and interactive 3D applications, the demand for high-quality modeling and flexible editing of 3D scenes is growing. Compared to building a complete 3D scene from scratch, users often prefer to use existing 3D scenes as templates for personalized modifications in the actual content creation process, thereby saving modeling costs and improving content generation efficiency. However, traditional methods often lack a sufficient understanding of 3D structure in texture modification, resulting in significant deficiencies in spatial structure and texture consistency in the edited results.

[0003] In actual texture reshaping, it is usually necessary to render multiple images from different perspectives from the original 3D scene, perform batch texture reshaping on these images, and finally send the editing results back to the 3D scene for updating and optimization. However, this process faces three key technical challenges: (1) It is difficult to maintain the consistency of the original structure during texture reshaping, which can easily lead to distortion or local deformation of the geometric structure of the reconstructed scene, affecting the overall visibility and realism; (2) The texture editing results of images from different perspectives within the same batch are inconsistent, which can easily lead to obvious discontinuities or artifacts in the 3D fusion stage, destroying the unified visual experience under multiple perspectives; (3) There are style or detail differences in texture reshaping between different batches, which can lead to inconsistent performance of the 3D scene constructed in the final return under different perspectives, making it difficult to meet the requirements of realism and immersion. Although some existing methods have tried to introduce consistency constraints or local guidance mechanisms to alleviate the above-mentioned multi-perspective problems, they still show limitations such as insufficient consistency control and insufficient structural understanding in complex texture reshaping tasks, and innovative methods with more structural awareness are urgently needed to make breakthroughs. Summary of the Invention

[0004] The purpose of this invention is to address the problems existing in the prior art and to provide a structure-aware multi-view texture reshaping method and system. This method aims to leverage structural information within existing 3D scenes to link texture information across different viewpoints, thereby improving the spatial coordination and visual consistency of texture editing. This approach not only enhances the flexibility and accuracy of 3D scene editing but also provides a reliable technical foundation for the rapid generation of 3D content for large-scale real-world scenes.

[0005] The inventive concept of this invention is as follows: using multi-view rendered images of the original 3D scene as input data, the rendered images from multiple views are fed into a diffusion model and control network in batches to extract structural information from each view, thereby guiding a batch of latent variables to denoise and reshape the texture. For the same batch of rendered images, a noise sharing module is introduced to controllably weight and mix the predicted noise within the batch, allowing each latent variable within the batch to obtain texture information from other latent variables. For different batches of rendered images, a dual-stream residual guidance module is introduced. By constructing a dual-branch architecture of reconstruction path and editing path, structural information is used as anchor points to guide the optimization process of the edited images, alleviating the problem of information isolation between different batches and providing a stable geometric semantic foundation for subsequent unified texture reshaping. After all batches of images have been edited, they can be used to optimize the original 3D scene to reshape its texture. This invention can efficiently complete customized texture editing of complex 3D scenes without relying on additional training, significantly improving the flexibility and quality of 3D content creation.

[0006] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0007] In a first aspect, the present invention provides a structure-aware multi-view texture reshaping method, which includes the following steps:

[0008] S1: Use a pre-trained diffusion model combined with a control network to process the multi-view rendering images of the original 3D scene in batches. Encode the multi-view rendering images using the image encoder in the diffusion model. Encode the text descriptions of the original scene and the target scene respectively through the text encoder in the diffusion model.

[0009] S2: Extract the structural information of the multi-view rendered image under each view, use the extracted structural information as structural guidance conditions, and pass it to each stage of the UNet network in the diffusion model through the cross-attention mechanism by the control network;

[0010] S3: Construct a dual-branch architecture for reconstruction and editing paths. Sample a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images.

[0011] S4: After all batches of multi-view rendered images have been edited, the resulting edited images are used to optimize the original 3D scene, achieving consistent texture reshaping across multiple perspectives.

[0012] Based on the above scheme, each step can be implemented in the following preferred manner.

[0013] As a preferred embodiment of the first aspect described above, in step S1, the control network includes a copy of the image encoder from the diffusion model and a zero convolutional layer.

[0014] As a preferred embodiment of the first aspect above, the specific steps in step S1 include:

[0015] S11: A batch of multi-view rendered images are sequentially fed into the image encoder of the diffusion model for processing to obtain the image code corresponding to each view of the rendered image.

[0016] S12: Encode the text descriptions of the original scene and the target scene using the text encoder in the diffusion model to obtain the text description encodings of the original scene and the target scene, respectively.

[0017] As a preferred embodiment of the first aspect, in step S2, the specific steps for extracting structural information are as follows: a batch of multi-view rendering images are sequentially fed into the structural feature extractor selected according to the three-dimensional scene for processing, so as to obtain the structural information corresponding to each viewpoint rendering image.

[0018] As a preferred embodiment of the first aspect above, the specific processing procedure in the noise sharing module in step S3 includes:

[0019] AS31: Under a standard Gaussian distribution, a batch of noise is sampled and used as the initial noise latent variable for the current batch. The same noise is copied and used as the initial noise latent variable for reconstructing the path.

[0020] AS32: Perform T-step iterative denoising on the reconstruction path using the initial noise latent variables of the reconstruction path, and perform T-step iterative denoising on the editing path using the initial noise latent variables obtained from sampling.

[0021] AS33: For the reconstruction path, the noise latent variable at the t-th time step is used as the reconstruction noise latent variable, and the current time step t, the reconstruction noise latent variable, the structural information of the current batch, and the text description encoding of the original scene are input into the UNet network to predict the noise; for the editing path, the noise latent variable at the t-th time step is used as the editing noise latent variable, and the current time step t, the editing noise latent variable, the structural information of the current batch, and the text description encoding of the target scene are input into the UNet network to predict the noise.

[0022] AS34: A shared time step threshold range is preset. If the current time step t is not within the shared time step threshold range, noise sharing is not performed, and the prediction noise corresponding to each viewpoint rendered image in the current batch is used directly. If the current time step t is within the shared time step threshold range, the prediction noise corresponding to each viewpoint rendered image in the current batch is traversed and weighted for sharing.

[0023] As a preferred embodiment of the first aspect above, in step AS34, the prediction noise of the rendered image at the t-th time step and the v-th viewpoint of the reconstruction path is used as the first prediction noise, the average value of the prediction noise of other rendered images in the current batch is used as the first auxiliary information, and the first auxiliary information and the first prediction noise are weighted to form the reconstruction shared noise of the rendered image.

[0024] The predicted noise of the rendered image at the t-th time step and the v-th viewpoint in the editing path is used as the second predicted noise. The average value of the predicted noise of other rendered images in the current batch is used as the second auxiliary information. The second auxiliary information and the second predicted noise are weighted together to form the editing shared noise of the rendered image.

[0025] Furthermore, in step AS34, the first The first time step Shared noise in the reconstruction of images rendered from different viewpoints Shared noise with editors The specific calculation method is as follows:

[0026]

[0027]

[0028] in, It refers to the batch size. It is a shared noise intensity; Indicates the reconstruction path number The first time step Predictive noise in images rendered from each viewpoint; Indicates the edit path number The first time step Predictive noise in images rendered from each viewpoint; Indicates the reconstruction path number The first time step Predictive noise in images rendered from each viewpoint; Indicates the edit path number The first time step Predicted noise in the rendered image from each viewpoint.

[0029] As a preferred embodiment of the first aspect mentioned above, the specific processing steps in the dual-stream residual guidance module in step S3 include:

[0030] BS31: For the t-th time step of the reconstruction path, the image encoding of the rendered image is multiplied by a preset first scaling factor to obtain a first calculation result. The reconstruction noise latent variable at this time step is subtracted from the first calculation result to obtain a second calculation result. The reciprocal of the preset second scaling factor is multiplied by the second calculation result to obtain the ideal noise at this time step. Wherein, the first scaling factor is the arithmetic square root of the guiding coefficient at the t-th time step of the reconstruction path, and the second scaling factor is the arithmetic square root of the difference between 1 and the guiding coefficient at the t-th time step of the reconstruction path.

[0031] BS32: Subtract the ideal noise at time step t from the shared noise of the reconstructed image in the rendered image, and use the result as the noise residual at that time step;

[0032] BS33: Use the ideal noise at time step t to denoise the latent noise variables of the reconstruction path at that time step, and generate the latent noise variables of the reconstruction at time step t-1.

[0033] BS34: For the t-th time step of the edit path, the noise residual is used as the anchor guide and multiplied by the guide intensity factor to form the noise correction amount. The noise correction amount is added to the edit-shared noise of the rendered image at that time step to form the noise of the edit path after correction at that time step.

[0034] BS35: Use the corrected noise at time step t to denoise the latent editing noise at that time step in the editing path, and generate the latent editing noise at time step t-1.

[0035] Furthermore, in step BS31, the path is reconstructed. Ideal noise at each time step The calculation method is as follows:

[0036]

[0037] in, Indicates the first proportionality coefficient; This is the second proportionality coefficient; It is the first The guiding coefficient at each time step; The image encoding that represents the rendered image; Indicates the first Reconstruction noise latent variables at each time step.

[0038] As a preferred embodiment of the first aspect mentioned above, the specific processing procedure for the denoising operation in step BS33 is as follows: multiply the ideal noise at the t-th time step by the second proportional coefficient to obtain the third calculation result; subtract the latent variable of the reconstructed noise at the t-th time step from the third calculation result to obtain the fourth calculation result; divide the fourth calculation result by the first proportional coefficient to obtain the fifth calculation result; multiply the fifth calculation result by the preset third proportional coefficient to obtain the sixth calculation result; multiply the ideal noise at the t-th time step by the preset fourth proportional coefficient to obtain the seventh calculation result; and add the sixth and seventh calculation results as the latent variable of the reconstructed noise at the (t-1)-th time step; wherein, the third proportional coefficient is the arithmetic square root of the guiding coefficient of the reconstruction path at the (t-1)-th time step, and the fourth proportional coefficient is the arithmetic square root of the difference between 1 and the guiding coefficient of the reconstruction path at the (t-1)-th time step.

[0039] Furthermore, in step BS33, the first Reconstruction noise latent variables at each time step The specific calculation method is as follows:

[0040]

[0041] in, Indicates the third proportionality coefficient; It is the fourth proportionality coefficient; It is the first The guiding coefficient at each time step.

[0042] As a preferred embodiment of the first aspect, the specific processing procedure for the denoising operation in step BS35 is as follows: multiply the corrected noise at the t-th time step by the second proportional coefficient to obtain the eighth calculation result; subtract the editing noise latent variable at the t-th time step from the eighth calculation result to obtain the ninth calculation result; divide the ninth calculation result by the first proportional coefficient to obtain the tenth calculation result; multiply the tenth calculation result by the third proportional coefficient to obtain the eleventh calculation result; multiply the corrected noise at the t-th time step by the fourth proportional coefficient to obtain the twelfth calculation result; and add the eleventh and twelfth calculation results together as the editing noise latent variable at the (t-1)-th time step.

[0043] Furthermore, in step BS35, the first Editing noise latent variables at each time step The specific calculation method is as follows:

[0044]

[0045]

[0046] in, Indicates the first Hidden variables of editing noise at each time step; Indicates the edit path is at the ___ Noise corrected at each time step; It is a guidance intensity factor used to balance the degree of freedom in reconstruction guidance and editing; Indicates the first Noise residual at each time step; Indicates the first Editing shared noise in the rendered image at each time step.

[0047] Secondly, the present invention provides a structure-aware multi-view texture reshaping system, comprising:

[0048] The encoding module is used to process multi-view rendered images of the original 3D scene in batches using a pre-trained diffusion model combined with a control network, encode the multi-view rendered images using the image encoder in the diffusion model, and encode the text descriptions of the original scene and the target scene respectively through the text encoder in the diffusion model.

[0049] The information injection module is used to extract structural information of the multi-view rendered image under each view. The extracted structural information is used as structural guidance conditions and is passed by the control network to each stage of the UNet network in the diffusion model through the cross attention mechanism.

[0050] The denoising module constructs a dual-branch architecture for the reconstruction path and the editing path. It samples a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images.

[0051] The result acquisition module is used to optimize the original 3D scene with the obtained edited image after all batches of multi-view rendered images have been edited, so as to achieve consistent texture reshaping from multiple perspectives.

[0052] Compared with the prior art, the present invention has the following advantages:

[0053] This invention introduces a structure-aware multi-view texture reshaping framework. Based on a pre-trained diffusion model, it constructs a collaborative mechanism of intra-batch sharing and inter-batch guidance to achieve consistent texture editing across views and batches. Specifically, during texture reshaping, multi-view images are input into the diffusion model in batches, and a noise sharing module is introduced. Within the same batch, predicted noise between views is controlled and weighted to enhance the transferability of texture features and improve the consistency of views within the same batch. Between batches, a dual-stream residual guidance module with reconstruction-editing dual paths is designed. Using structural information as an anchor, it guides the latent variable optimization process in the editing path, thereby alleviating the information breakage problem between different batches and ensuring the continuity of geometric structure and semantic expression. Finally, the edited image can be fed back to the original 3D scene, achieving structurally consistent and texture-unified multi-view scene reconstruction. Based on this invention, users can efficiently complete customized texture editing of complex 3D scenes without relying on additional training, significantly improving the flexibility and quality of 3D content creation. Attached Figure Description

[0054] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0055] Figure 2 This is a schematic diagram illustrating the process of reshaping textures using the method of the present invention;

[0056] Figure 3 This is a comparison image of the editing results of the present invention and other texture reshaping methods from different perspectives;

[0057] Figure 4 This is a system block diagram of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0059] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0060] In this embodiment, the technical background of the present invention will first be described, which can be summarized in the following aspects:

[0061] 1) Texture Reshaping: This refers to the process of replacing the style, changing the material, or enhancing the semantics of an object's surface texture while maintaining the stability of the 3D structure. This process typically involves rendering multiple viewpoint images from an existing 3D scene, then modifying the texture locally or globally using an image editing model, and finally projecting the edited results back into 3D space to update the scene. This method offers advantages such as high flexibility and efficiency, but it also places high demands on the alignment between texture and structure during the editing process. If the texture modification exceeds the structural boundaries or lacks semantic continuity, it will directly affect the realism of the edited result and the consistency of the scene. Therefore, texture reshaping methods must be combined with structural constraints to precisely control the modification range and ensure that the modified texture maintains a consistent appearance under different viewpoints.

[0062] 2) Structural and Texture Consistency of Multi-View Images: In 3D scene editing tasks, multi-view consistency is a core element affecting the final reconstruction quality. Structural consistency requires that targets in all views perfectly match in terms of geometric contours and spatial positions; texture consistency requires that texture details and styles remain consistent and seamless across different views. Since multi-view images are processed in batches, there is often a lack of corresponding connections and constraints within and between batches during texture reshaping, leading to structural distortion, texture shifts, or semantic conflicts in the generated results across different views. To address this issue, texture reshaping methods need to establish cross-view alignment mechanisms and use structural information as anchor points to guide images from different views to remain synchronized during the editing process. Furthermore, transition consistency between batches should be considered to prevent style abrupt changes or texture breaks during multi-stage processing. Only by achieving high consistency across multiple views in both structural and texture dimensions can the reconstructed 3D scene be guaranteed to have a stable and realistic appearance.

[0063] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned structure-aware multi-view texture reshaping method includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0064] S1: Use a pre-trained diffusion model combined with a control network to process the multi-view rendering images of the original 3D scene in batches. Encode the multi-view rendering images using the image encoder in the diffusion model. Encode the text descriptions of the original scene and the target scene using the text encoder in the diffusion model for subsequent use.

[0065] S2: Extract structural information of the multi-view rendered image at each viewpoint, use the extracted structural information as structural guidance conditions, and pass it to each stage of the UNet network in the diffusion model through the control network via the cross-attention mechanism.

[0066] S3: Construct a dual-branch architecture for reconstruction and editing paths. Sample a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images.

[0067] S4: After all batches of multi-view rendered images have been edited, the resulting edited images are used to optimize the original 3D scene, achieving consistent texture reshaping across multiple perspectives.

[0068] The method of the present invention described in S1 to S4 above can efficiently complete customized texture editing of complex 3D scenes without relying on additional training in the task of multi-view texture reshaping based on structure awareness, and significantly improve the flexibility and quality of 3D content creation.

[0069] In this embodiment, the above steps can be implemented in the following specific ways.

[0070] It should be noted that in step S1 of the present invention, the diffusion model includes a text encoder, an image encoder and a U-shaped network (i.e., UNet). This model is pre-trained on a large amount of text image data and has a super image generation capability.

[0071] The structure-aware control network aims to provide additional, refined conditional inputs and control to the diffusion model. It comprises a trainable copy of the image encoder from the diffusion model and zero-convolutional layers, capable of receiving additional structure-guided conditional inputs (such as edge maps, depth maps, or normal maps). This structural information is embedded as constraint signals into the image generation and editing process, ensuring that the geometric contours and spatial layout of the target are preserved when modifying textures, thus achieving precise generational control over the pre-trained diffusion model. Specifically, this network introduces an auxiliary branch outside the main diffusion model, receiving the structure-guided map and extracting its high-level semantic and spatial features. These features are then passed to each stage of the main network through a cross-layer fusion mechanism. During training, the main model parameters remain frozen, and the control network influences the generation process through fine-tuning, thereby achieving high-fidelity and high-consistency image editing. In 3D scene texture reshaping tasks, the control network effectively avoids structural deformation or spatial misalignment while ensuring accurate texture modification, making it a key module for ensuring consistency in multi-view editing.

[0072] It should be noted that, as Figure 2 As shown, the specific steps of S1 include:

[0073] S11: A batch of multi-view rendered images are sequentially fed into the image encoder of the diffusion model for processing, resulting in the image code corresponding to each viewpoint of the rendered image.

[0074] In this embodiment, for the rendered image of the current viewpoint in this batch Its corresponding image encoding is , It is an image encoder.

[0075] S12: Text description of the original scene and text description of the target scene Using the text encoder in the diffusion model respectively The encoding corresponds to the text description encoding of the original scene. Text description encoding of the target scene :

[0076]

[0077]

[0078] The image encoding above stores the structural and texture information of each viewpoint, while the text description encoding stores the semantic features of the original scene and the target scene. This lays an important foundation for achieving accurate and consistent texture reshaping across multiple viewpoints.

[0079] It should be noted that in this embodiment, step S2 aims to maintain the consistency of the original structure during texture reshaping. The specific steps for extracting structural information are as follows: A suitable structural feature extractor (such as depth, normals, edges, etc.) is selected based on the 3D scene. A batch of multi-view rendered images is sequentially fed into the selected structural feature extractor for processing, obtaining the structural information corresponding to each viewpoint's rendered image. For the rendered image of the current viewpoint in this batch... Its corresponding structural information is , It is the selected structural feature extractor.

[0080] It should be noted that, in this embodiment, the noise sharing module in step S3 aims to prevent inconsistent texture editing results between rendered images from different perspectives within the same batch. It performs controllable weighted mixing of the predicted noise corresponding to the multi-view rendered images in the same batch, enabling the noise latent variables at each time step within the batch to obtain the texture information of other noise latent variables at that time step. The dual-stream residual guidance module aims to alleviate the problem of information isolation between different batches. By constructing a dual-branch architecture of reconstruction and editing paths, it uses structural information as anchor points to guide the optimization process of the edited images, providing a stable geometric semantic foundation for subsequent unified texture reshaping.

[0081] It should be noted that the specific processing steps in step S3 of the noise sharing module include:

[0082] AS31: Under a standard Gaussian distribution, sample a batch of noise as the initial noise latent variable for the current batch. The same noise is replicated for the initial noise latent variables used to reconstruct the path. .in, Represents a standard Gaussian distribution. Represents the identity matrix.

[0083] AS32: Using initial noise latent variables On the reconstruction path Step-by-step iterative denoising, using initial noise latent variables On the editing path Iterative noise reduction. This represents the total time steps of the denoising process.

[0084] AS33: For the reconstruction path, the first... The noise latent variables at each time step are used to reconstruct the noise latent variables. , set the current time step The reconstructed noise latent variables, the structural information of the current batch, and the text description encoding of the original scene are input into the UNet network to predict the noise. For editing paths, the first... The noise latent variables at each time step are used as editing noise latent variables. , set the current time step The noise latent variables, structural information of the current batch, and text description encoding of the target scene are input into the UNet network to predict noise. .

[0085] AS34: Presets a shared time step threshold range, if the current time step If the noise is not within the shared time step threshold, then noise sharing is not performed, and the predicted noise corresponding to each viewpoint rendered image in the current batch is used directly; if the current time step... Within the shared time step threshold, the predicted noise corresponding to each viewpoint rendered image in the current batch is weighted and shared.

[0086] In this invention, the reconstruction path is... The first time step Predictive noise in image rendering from each viewpoint As the first prediction noise, the average value of the prediction noise of other viewpoint rendered images in the current batch is used as the first auxiliary information. The first auxiliary information is weighted with the first prediction noise to form the reconstruction shared noise of the rendered image.

[0087] In this invention, the editing path is... The first time step Predictive noise in image rendering from each viewpoint As a second prediction noise, the average value of the prediction noise of other viewpoint rendered images in the current batch is used as the second auxiliary information. The second auxiliary information is weighted with the second prediction noise to form the edit-shared noise of the rendered image.

[0088] In this embodiment, the reconstruction of the rendered image shares noise. Shared noise with editors The specific calculation method is as follows:

[0089]

[0090]

[0091] in, It refers to the batch size. It is a shared noise intensity; Indicates the reconstruction path number The first time step Predictive noise in images rendered from each viewpoint; Indicates the edit path number The first time step Predicted noise in the rendered image from each viewpoint.

[0092] The aforementioned noise sharing module introduces noise information from samples from other perspectives in the early stages of diffusion, enabling each sample to have a stronger global context awareness, thereby effectively improving texture consistency. In the later stages, when noise sharing stops, the unique details of each perspective are preserved, achieving a balance between global consistency and local fidelity.

[0093] It should be noted that the specific processing steps in step S3 of the dual-stream residual guidance module include:

[0094] BS31: For the first reconstruction path Each time step encodes the image of the rendered image. and the preset first proportional coefficient Multiplying them yields the first calculation result, which represents the latent reconstruction noise variable at that time step. Subtracting the first calculation result from the second calculation result, the reciprocal of the preset second proportional coefficient is multiplied by the second calculation result to obtain the ideal noise at that time step. ; where the first proportionality coefficient is the reconstruction path number ; The arithmetic square root of the guiding coefficient at the nth time step, the second proportionality coefficient being 1 and the reconstruction path at the nth time step. The arithmetic square root of the difference between the guiding coefficients at each time step.

[0095] In this embodiment, the reconstruction path is... Ideal noise at each time step The calculation method is as follows:

[0096]

[0097] in, Indicates the first proportionality coefficient; This is the second proportionality coefficient; It is the first The guiding coefficient at each time step.

[0098] BS32: The first The ideal noise at each time step and the reconstructed noise of the rendered image share noise. Subtract from each other to obtain the noise residual at that time step. .

[0099] BS33: Using the first Ideal noise at each time step Reconstruction noise latent variables at this time step in the reconstruction path Perform denoising operation to generate the first Reconstruction noise latent variables at each time step .

[0100] It should be noted that in step BS33 of this invention, the specific processing procedure for the noise reduction operation is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Ideal noise at each time step Multiply by the second proportionality coefficient to obtain the third calculation result, and then... Reconstruction noise latent variables at each time step Subtracting the third calculation result yields the fourth calculation result. Dividing the fourth calculation result by the first proportionality coefficient yields the fifth calculation result. Multiplying the fifth calculation result by the preset third proportionality coefficient yields the sixth calculation result. Ideal noise at each time step Multiply by the preset fourth proportional coefficient to obtain the seventh calculation result. Add the sixth and seventh calculation results to obtain the sixth result. Reconstruction noise latent variables at each time step Among them, the third proportionality coefficient is the reconstruction path number. The arithmetic square root of the guiding coefficient at the nth time step, the fourth proportionality coefficient is 1 and the reconstruction path is nth The arithmetic square root of the difference between the guiding coefficients at each time step.

[0101] In this embodiment, the first Reconstruction noise latent variables at each time step The specific calculation method is as follows:

[0102]

[0103] in, Indicates the third proportionality coefficient; It is the fourth proportionality coefficient; It is the first The guiding coefficient at each time step.

[0104] BS34: For the first edit path Each time step will include the noise residual. As an anchor guide, it is multiplied by the guide strength factor to form a noise correction amount. This noise correction amount is then shared with the edit noise of the rendered image at that time step. The addition forms the noise of the edit path corrected at that time step. .

[0105] In this embodiment, the edit path is at the 1st Noise corrected at each time step The calculation method is as follows:

[0106]

[0107] in, It is the guidance intensity factor, used to balance the degree of freedom of reconstruction guidance and editing.

[0108] BS35: Using the first Noise corrected at each time step Latent variables of editing noise at this time step in the editing path Perform denoising operation to generate the first Editing noise latent variables at each time step .

[0109] It should be noted that in step BS35 of this invention, the specific processing procedure for the noise reduction operation is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Noise corrected at each time step Multiply by the second proportionality coefficient to obtain the eighth calculation result, and then... Editing noise latent variables at each time step Subtracting the eighth calculation result yields the ninth calculation result. Dividing the ninth calculation result by the first proportionality coefficient yields the tenth calculation result. Multiplying the tenth calculation result by the third proportionality coefficient yields the eleventh calculation result. Noise corrected at each time step Multiply by the fourth proportionality coefficient to obtain the twelfth calculation result. Add the eleventh and twelfth calculation results together to obtain the eleventh result. Editing noise latent variables at each time step .

[0110] In this embodiment, the first Editing noise latent variables at each time step The specific calculation method is as follows:

[0111]

[0112] The aforementioned dual-stream residual guidance module effectively transmits the deviation signals of UNet's understanding of structure and texture from different viewpoints from the reconstruction path to the editing path, alleviating the problems of texture offset and uneven editing intensity caused by viewpoint differences during the editing process, thereby improving the consistency and coordination of cross-viewpoint editing results in terms of texture expression and structure preservation.

[0113] The structure-aware multi-view texture reshaping method described in steps S1-S4 of the above preferred implementation will be applied to a specific dataset for testing. The specific steps are as described in S1-S4 and will not be repeated here. The main focus is on demonstrating the specific parameters and technical effects.

[0114] Example

[0115] Following the implementation process of steps S1-S4 described above, a 3D scene dataset collected from the real world is first acquired for 3D scene reconstruction, followed by rendering of multi-view images. During texture reshaping, the multi-view rendered images are input into a diffusion model in batches, and a noise sharing module is introduced. Within the same batch, the predicted noise between views is controlled and weighted to improve the transferability of texture features and enhance the consistency of views within the same batch. Between batches, a dual-stream residual guidance module with reconstruction-editing dual paths is designed. Using structural information as an anchor point, it guides the latent variable optimization process in the editing path, thereby alleviating the information breakage problem between different batches and ensuring the continuity of geometric structure and semantic expression. Finally, the edited image can be fed back to the original 3D scene, achieving a multi-view scene reconstruction with consistent structure and unified texture.

[0116] To quantify the metrics, this embodiment compares the method of the present invention with four existing methods: IN2N(GS), GaussianEditor, GaussCtrl, and DGE. Three metrics are selected: CLIP text-image orientation similarity, CLIP orientation consistency, and user preference. The test results of the above methods on the IN2N dataset are shown in Table 1. The editing effects of the method of the present invention and the four existing methods are compared as follows: Figure 3 As shown.

[0117] Table 1. Quantitative Evaluation Table of Editing Effectiveness

[0118]

[0119] It should also be noted that the structure-aware multi-view texture reshaping method in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a structure-aware multi-view texture reshaping system corresponding to the structure-aware multi-view texture reshaping method provided in the above embodiments, such as... Figure 4 As shown, it includes:

[0120] The encoding module is used to process multi-view rendered images of the original 3D scene in batches using a pre-trained diffusion model combined with a control network, encode the multi-view rendered images using the image encoder in the diffusion model, and encode the text descriptions of the original scene and the target scene respectively through the text encoder in the diffusion model.

[0121] The information injection module is used to extract structural information of the multi-view rendered image under each view. The extracted structural information is used as structural guidance conditions and is passed by the control network to each stage of the UNet network in the diffusion model through the cross attention mechanism.

[0122] The denoising module constructs a dual-branch architecture for the reconstruction path and the editing path. It samples a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images.

[0123] The result acquisition module is used to optimize the original 3D scene with the obtained edited image after all batches of multi-view rendered images have been edited, so as to achieve consistent texture reshaping from multiple perspectives.

[0124] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0125] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A structure-aware multi-view texture reshaping method, characterized in that, Includes the following steps: S1: Use a pre-trained diffusion model combined with a control network to process the multi-view rendering images of the original 3D scene in batches. Encode the multi-view rendering images using the image encoder in the diffusion model. Encode the text descriptions of the original scene and the target scene respectively through the text encoder in the diffusion model. S2: Extract the structural information of the multi-view rendered image under each view, use the extracted structural information as structural guidance conditions, and pass it to each stage of the UNet network in the diffusion model through the cross-attention mechanism by the control network; S3: Construct a dual-branch architecture for reconstruction and editing paths. Sample a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images. S4: After all batches of multi-view rendered images have been edited, the original 3D scene is optimized using the resulting edited images to achieve consistent texture reshaping across multiple perspectives; In step S3, the specific processing steps in the dual-stream residual guidance module include: BS31: For the t-th time step of the reconstruction path, the image encoding of the rendered image is multiplied by a preset first scaling factor to obtain a first calculation result. The reconstruction noise latent variable at this time step is subtracted from the first calculation result to obtain a second calculation result. The reciprocal of the preset second scaling factor is multiplied by the second calculation result to obtain the ideal noise at this time step. Wherein, the first scaling factor is the arithmetic square root of the guiding coefficient at the t-th time step of the reconstruction path, and the second scaling factor is the arithmetic square root of the difference between 1 and the guiding coefficient at the t-th time step of the reconstruction path. BS32: Subtract the ideal noise at time step t from the shared noise of the reconstructed image in the rendered image, and use the result as the noise residual at that time step; BS33: Use the ideal noise at time step t to denoise the latent noise variables of the reconstruction path at that time step, and generate the latent noise variables of the reconstruction at time step t-1. BS34: For the t-th time step of the edit path, the noise residual is used as the anchor guide and multiplied by the guide intensity factor to form the noise correction amount. The noise correction amount is added to the edit-shared noise of the rendered image at that time step to form the noise of the edit path after correction at that time step. BS35: Use the corrected noise at time step t to denoise the latent editing noise at that time step in the editing path, and generate the latent editing noise at time step t-1.

2. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, In step S1, the control network includes a copy of the image encoder from the diffusion model and a zero convolutional layer.

3. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, The specific steps in step S1 include: S11: A batch of multi-view rendered images are sequentially fed into the image encoder of the diffusion model for processing to obtain the image code corresponding to each view of the rendered image. S12: Encode the text descriptions of the original scene and the target scene using the text encoder in the diffusion model to obtain the text description encodings of the original scene and the target scene, respectively.

4. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, The specific steps for extracting structural information in step S2 are as follows: a batch of multi-view rendered images are sequentially fed into the structural feature extractor selected according to the 3D scene for processing, so as to obtain the structural information corresponding to each view of the rendered image.

5. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, In step S3, the specific processing steps in the noise sharing module include: AS31: Under a standard Gaussian distribution, a batch of noise is sampled and used as the initial noise latent variable for the current batch. The same noise is copied and used as the initial noise latent variable for reconstructing the path. AS32: Perform T-step iterative denoising on the reconstruction path using the initial noise latent variables of the reconstruction path, and perform T-step iterative denoising on the editing path using the initial noise latent variables obtained from sampling. AS33: For the reconstruction path, the noise latent variable at the t-th time step is used as the reconstruction noise latent variable, and the current time step t, the reconstruction noise latent variable, the structural information of the current batch, and the text description encoding of the original scene are input into the UNet network to predict the noise; for the editing path, the noise latent variable at the t-th time step is used as the editing noise latent variable, and the current time step t, the editing noise latent variable, the structural information of the current batch, and the text description encoding of the target scene are input into the UNet network to predict the noise. AS34: A shared time step threshold range is preset. If the current time step t is not within the shared time step threshold range, noise sharing is not performed, and the prediction noise corresponding to each viewpoint rendered image in the current batch is used directly. If the current time step t is within the shared time step threshold range, the prediction noise corresponding to each viewpoint rendered image in the current batch is traversed and weighted for sharing.

6. The structure-aware multi-view texture reshaping method as described in claim 5, characterized in that, In step AS34, the predicted noise of the rendered image at the t-th time step and the v-th viewpoint of the reconstruction path is used as the first predicted noise, and the average value of the predicted noise of other rendered images in the current batch is used as the first auxiliary information. The first auxiliary information and the first predicted noise are weighted to form the reconstruction shared noise of the rendered image. The predicted noise of the rendered image at the t-th time step and the v-th viewpoint in the editing path is used as the second predicted noise. The average value of the predicted noise of other rendered images in the current batch is used as the second auxiliary information. The second auxiliary information and the second predicted noise are weighted together to form the editing shared noise of the rendered image.

7. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, The specific processing steps of the denoising operation in step BS33 are as follows: multiply the ideal noise at time step t by the second proportional coefficient to obtain the third calculation result; subtract the latent variable of the reconstructed noise at time step t from the third calculation result to obtain the fourth calculation result; divide the fourth calculation result by the first proportional coefficient to obtain the fifth calculation result; multiply the fifth calculation result by the preset third proportional coefficient to obtain the sixth calculation result; multiply the ideal noise at time step t by the preset fourth proportional coefficient to obtain the seventh calculation result; and add the sixth and seventh calculation results as the latent variable of the reconstructed noise at time step t-1. Among these, the third proportional coefficient is the arithmetic square root of the guiding coefficient of the reconstruction path at time step t-1, and the fourth proportional coefficient is the arithmetic square root of the difference between 1 and the guiding coefficient of the reconstruction path at time step t-1.

8. The structure-aware multi-view texture reshaping method as described in claim 1, characterized in that, The specific processing steps for the denoising operation in step BS35 are as follows: multiply the corrected noise at time step t by the second proportional coefficient to obtain the eighth calculation result; subtract the editing noise latent variable at time step t from the eighth calculation result to obtain the ninth calculation result; divide the ninth calculation result by the first proportional coefficient to obtain the tenth calculation result; multiply the tenth calculation result by the third proportional coefficient to obtain the eleventh calculation result; multiply the corrected noise at time step t by the fourth proportional coefficient to obtain the twelfth calculation result; and add the eleventh and twelfth calculation results together as the editing noise latent variable at time step t-1.

9. A structure-aware multi-view texture reshaping system, characterized in that, include: The encoding module is used to process multi-view rendered images of the original 3D scene in batches using a pre-trained diffusion model combined with a control network, encode the multi-view rendered images using the image encoder in the diffusion model, and encode the text descriptions of the original scene and the target scene respectively through the text encoder in the diffusion model. The information injection module is used to extract structural information of the multi-view rendered image under each view. The extracted structural information is used as structural guidance conditions and is passed by the control network to each stage of the UNet network in the diffusion model through the cross attention mechanism. The denoising module constructs a dual-branch architecture for the reconstruction path and the editing path. It samples a batch of noise under a standard Gaussian distribution as the initial noise latent variables for both the editing and reconstruction paths. The UNet network for the reconstruction path takes the initial noise latent variables, the text description encoding of the original scene, and the extracted structural information as input. The UNet network for the editing path takes the initial noise latent variables, the text description encoding of the target scene, and the extracted structural information as input. Both paths undergo multi-step iterative denoising. During the iterative denoising process, the noise sharing module performs controllable weighted mixing of the predicted noise corresponding to the same batch of multi-view rendered images, and the dual-stream residual guidance module corrects the predicted noise generated by the editing path for different batches of multi-view rendered images. The result acquisition module is used to optimize the original 3D scene with the obtained edited image after all batches of multi-view rendered images have been edited, so as to achieve consistent texture reshaping from multiple perspectives. In the denoising module, the specific processing steps in the dual-stream residual guidance module include: BS31: For the t-th time step of the reconstruction path, the image encoding of the rendered image is multiplied by a preset first scaling factor to obtain a first calculation result. The reconstruction noise latent variable at this time step is subtracted from the first calculation result to obtain a second calculation result. The reciprocal of the preset second scaling factor is multiplied by the second calculation result to obtain the ideal noise at this time step. Wherein, the first scaling factor is the arithmetic square root of the guiding coefficient at the t-th time step of the reconstruction path, and the second scaling factor is the arithmetic square root of the difference between 1 and the guiding coefficient at the t-th time step of the reconstruction path. BS32: Subtract the ideal noise at time step t from the shared noise of the reconstructed image in the rendered image, and use the result as the noise residual at that time step; BS33: Use the ideal noise at time step t to denoise the latent noise variables of the reconstruction path at that time step, and generate the latent noise variables of the reconstruction at time step t-1. BS34: For the t-th time step of the edit path, the noise residual is used as the anchor guide and multiplied by the guide intensity factor to form the noise correction amount. The noise correction amount is added to the edit-shared noise of the rendered image at that time step to form the noise of the edit path after correction at that time step. BS35: Use the corrected noise at time step t to denoise the latent editing noise at that time step in the editing path, and generate the latent editing noise at time step t-1.

Images