Graph-to-graph conversion method and apparatus using noise control module and diffusion model principles for fine tuning

Abstract

The application discloses a kind of graph-graph conversion method and device for fine tuning using noise control module and diffusion model principle, belong to computer vision technical field, including: original image and target image are compressed into original latent variable and target latent variable;Original latent variable is input into two noise control modules, and standard Gaussian noise is scaled and offset to generate actual noise, for the original latent variable is added to obtain noisy latent variable, and the pre-trained diffusion model is used to denoise noisy latent variable;Freeze the parameters of pre-trained diffusion model and the parameters of two noise control modules in turn for alternating training and fine tuning;For new input image, use the two noise control modules trained to partially add noise, and then use the fine-tuned diffusion model to denoise, restore to image space to obtain generated image.The application can significantly improve the detail retention capability of image generation, thereby efficiently and high-qualityly completing image restoration and style transfer and the like tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an image-to-image conversion method and apparatus that utilizes a noise control module and diffusion model principle for fine-tuning. Background Technology

[0002] In recent years, diffusion models, as an important type of generative model, have been widely applied to tasks such as image generation. Their principles include forward noise addition (adding noise to the image to corrupt the data) and backward denoising (learning to reconstruct data from noise), enabling the learning and generation of complex data distributions. During the training phase, the model learns how to recover an image from a noisy image by predicting known added noise; during the generation phase, it generates an image from the noisy image through iterative denoising in a certain way. Compared to traditional generative adversarial networks (GANs), diffusion models show significant advantages in training stability, generation diversity, and image quality, especially in generating high-resolution, detailed images.

[0003] However, traditional diffusion models still have certain limitations when applied to image generation tasks. On the one hand, their training process is often based on a single dataset; in other words, they only learn the probability distribution of images in the same dataset to generate images similar to that dataset, and cannot perform rule-based image-to-image transformations. This means that if we want to achieve image-to-image transformation tasks such as image inpainting and style transfer, traditional models often struggle to provide effective guidance without introducing additional mechanisms. On the other hand, traditional diffusion models typically use standard Gaussian noise during the noise addition process. This form of noise lacks semantic directionality and fails to guide the transformation towards the target image. Therefore, in tasks that require inferring missing or damaged parts based on the original image structure (such as inpainting tasks), the direct noise addition-denoising approach struggles to maintain semantic consistency and cannot achieve precise control over the restoration target.

[0004] These limitations are particularly prominent in the practical protection of cultural heritage, especially in the restoration of ancient murals. As important carriers of history and art, murals have endured hundreds or even thousands of years of natural weathering, human damage, and environmental erosion, resulting in a wide range of complex damage, including cracks, peeling, discoloration, and contamination. Image restoration technology aims to automatically or semi-automatically restore their original appearance. This requires not only that the technology reconstruct the visual structure of the missing areas but also that it maintain consistency with the original work in terms of semantics, texture, and style. This places extremely high demands on the cross-domain transformation capabilities and structural control of the generative model.

[0005] Therefore, despite the powerful generative capabilities demonstrated by diffusion models, their direct application to mural restoration still faces significant challenges: how to achieve visual rationality and artistic consistency in restoration without damaging the original structure remains a crucial issue that urgently needs to be addressed. Against this backdrop, effectively combining diffusion models with image transformation tasks to adapt them to generative scenarios based on conditional inputs (such as damaged images) has become one of the key research directions. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide an image-to-image conversion method and apparatus that utilizes a noise control module and a diffusion model principle for fine-tuning. By designing a noise control module for noise modulation and flexibly controlling standard Gaussian noise using scaling and offset parameters, the latent variable distribution after noise addition is made closer to the characteristics of the target image. At the same time, combined with a phased training strategy, the noise control module is first optimized to improve the accuracy of noise addition, and then the diffusion model is fine-tuned to enhance the denoising capability. Finally, the synergistic optimization of the noise addition and denoising process is achieved, thereby efficiently and with high quality completing tasks such as image restoration (e.g., mural restoration) and style transfer.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution: This invention provides a graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principles, comprising the following steps: The original image and the target image are compressed into the latent space to obtain the original latent variables and the target latent variables; The original latent variables are input into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate the actual noise. The original latent variables are then denoised using the actual noise to obtain the noisy latent variables. The noisy latent variables are then denoised using a pre-trained diffusion model to obtain the predicted latent variables. The parameters of the pre-trained diffusion model are frozen, and a loss function is constructed based on the predicted latent variables and the target latent variables to train the two noise control modules. Freeze the parameters of the two trained noise control modules, construct a loss function based on the actual predicted noise and the target predicted noise of the diffusion model, and use it to fine-tune the pre-trained diffusion model; For a new input image, two trained noise control modules are used to add noise to the image, and then a finely tuned diffusion model is used to remove noise, restoring the image to the image space to obtain the generated image.

[0008] Preferably, the step of inputting the original latent variables into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate the actual noise, includes: Using two original latent variables The first noise control module as input With the second noise control module Sampling standard Gaussian noise Calculate the actual noise : , Among them, based on The first modulation parameter is calculated and used for... Scaling, based on The calculated second modulation parameter is used for... Perform an offset.

[0009] Preferably, the step of adding noise to the original latent variable using actual noise to obtain a noisy latent variable includes: Time step The noise addition formula is: , in, Indicates noisy latent variables; , , , in, Indicates time step Noise control parameters at that time, Represents the cumulative noise figure. Indicates time step The noise variance coefficient at that time, and These represent the maximum noise variance and the minimum noise variance, respectively. This represents the total number of time steps.

[0010] Preferably, when generating actual noise, each The standard Gaussian noise of the sample is replaced step by step to construct a new distribution of the actual noise.

[0011] Preferably, the loss function constructed based on the actual predicted noise and the target predicted noise of the diffusion model includes: diffusion model The actual predicted noise in the denoising process is represented as: ,in Represents network parameters, Indicates the original latent variables The noisy latent variable is obtained by adding noise through two noise control modules; Target prediction noise Represented as: , in, Indicates time step The cumulative noise figure at that time Represents the target latent variable; The loss function for the fine-tuning process is expressed as: , in, This represents the mean squared error.

[0012] Preferably, both the first noise control module and the second noise control module are constructed based on a fully convolutional network.

[0013] Preferably, the diffusion model is constructed based on the U-Net network.

[0014] Preferably, a variational autoencoder is used for image compression and image restoration.

[0015] Secondly, embodiments of the present invention also provide an image-to-image conversion device that uses a noise control module and a diffusion model principle for fine-tuning, based on the above-mentioned image-to-image conversion method that uses a noise control module and a diffusion model principle for fine-tuning, including: an image compression module, a noise modulation module, a noise addition training module, a noise reduction fine-tuning module, and an image generation module; The image compression module is used to compress the original image and the target image into the latent space to obtain the original latent variables and the target latent variables; The noise modulation module is used to input the original latent variables into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate actual noise. The original latent variables are then denoised using the actual noise to obtain noisy latent variables. Finally, the noisy latent variables are denoised using a pre-trained diffusion model to obtain the predicted latent variables. The noise-adding training module is used to freeze the parameters of the pre-trained diffusion model and construct a loss function based on the predicted latent variables and the target latent variables, which is used to train the two noise control modules. The denoising fine-tuning module is used to freeze the parameters of the two trained noise control modules, and construct a loss function based on the actual predicted noise and target predicted noise of the diffusion model, which is used to fine-tune the pre-trained diffusion model. The image generation module is used to partially add noise to a new input image using two trained noise control modules, and then use a fine-tuned diffusion model to remove noise, restoring the image to the image space to obtain the generated image.

[0016] Thirdly, embodiments of the present invention also provide a method for mural restoration or style transfer, which is based on the above-mentioned image-to-image conversion method that uses a noise control module and diffusion model principle for fine-tuning, and is used to restore the damage to the mural or to generate an image for image style change.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: This invention designs two noise control modules to scale and offset standard Gaussian noise, respectively, making the latent variable distribution after noise addition closer to the characteristics of the target image. This effectively solves the problem of blurring or distortion caused by uncontrollable noise in traditional methods, significantly improving the detail preservation ability of image restoration (such as mural detail restoration) and style transfer. Furthermore, during training, a staged optimization approach is adopted: first freezing the diffusion model to train the noise module, then freezing the noise module to fine-tune the diffusion model. This avoids the high computational overhead of joint training of the entire model. Simultaneously, module decoupling improves model generalization, reducing training data requirements while maintaining generation quality, making it more suitable for resource-constrained practical applications. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principles, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the framework of the graph-to-graph conversion method for fine-tuning using a noise control module and the principle of a diffusion model, provided in an embodiment of the present invention. Figure 3 These are before-and-after comparison images of mural restoration provided in Embodiment 1 of the present invention; Figure 4 This is a comparison image of style transfer before and after provided in Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of the image-to-image conversion device that utilizes a noise control module and the principle of a diffusion model for fine-tuning, as provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0021] The inventive concept of this invention is as follows: Addressing the problem of low generation quality in existing image generation methods for tasks such as image restoration, particularly mural restoration, and style transfer, this invention provides an image-to-image conversion method and apparatus that utilizes a noise control module and a diffusion model principle for fine-tuning. By compressing the original image and the target image into a latent space, modulated actual noise is generated using first and second noise control modules, achieving precise noise addition to the original latent variables. Subsequently, a pre-trained diffusion model is used to denoise and obtain predicted latent variables. A staged training strategy is employed for model training and fine-tuning. This approach enhances the flexibility and controllability of the noise addition process through the noise control module and improves denoising accuracy through diffusion model fine-tuning, thereby efficiently and effectively completing tasks such as image restoration and style transfer, and improving generalization ability in practical applications.

[0022] Example 1 like Figure 1 and Figure 2 As shown, this embodiment provides a fine-tuning image-to-image conversion method using a noise control module and diffusion model principles, applied to mural restoration, including the following steps: S1, compress the original image and the target image into the latent space to obtain the original latent variables and the target latent variables.

[0023] In this embodiment, the original image and the target image are compressed into the latent space using a variational autoencoder (VAE) to obtain the original latent variables. and target latent variables .

[0024] Specifically, the VAE uses the AutoEncoderKL module from Stable Diffusion 1.5 to save computational resources. The original images consist of 44 original mural images, and the target images consist of 44 corresponding restored mural images, manually repaired to remove scratches, tears, spots, and other damage. During training, these images are randomly cropped to 512*512 pixels in RGB 3-channel format (if the length or width is less than 512, a square portion with the smaller length and width is cropped and enlarged to 512*512). Then, during training, the VAE compresses these images into the latent space to obtain 4*64*64 latent variables, thereby accelerating computation and saving GPU memory. It should be noted that practices using a similar process to S2-S5 but without VAE compression should be considered as employing the same method as this invention.

[0025] S2, the original latent variables are input into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate actual noise. The original latent variables are then denoised using the actual noise to obtain noisy latent variables. The noisy latent variables are then denoised using a pre-trained diffusion model to obtain the predicted latent variables.

[0026] Because the noise uses a standard Gaussian distribution, the image is uniformly blurred, making originally different local features more similar. This makes it more difficult for U-Net to learn how to process different local features differently during the subsequent fine-tuning. Therefore, this invention employs two noise control modules. and Therefore, it can be used The noise distribution is used to make the noisy image closer to the target image distribution, making it easier for the subsequent U-Net to learn how to use the noisy image to transform into the target image.

[0027] In this embodiment, a fully convolutional network (FCN) is used as the first noise control module and the second noise control module to process the original latent variables. Partial noise addition yields noisy latent variables. Then, a pre-trained diffusion model based on the U-Net network is used to analyze the noisy latent variables. Denoising is performed to obtain the predicted latent variables. .

[0028] Specifically, the FCN network uses the original semantic segmentation FCN network in PyTorch with ResNet50 as the backbone, followed by 5 convolutional layers, reducing the output channels from 21 to 4 to match the number of latent variable channels, and finally a Sigmoid activation function. U-Net uses the U-Net module from Stable Diffusion 1.5. It uses two channels with the original latent variables... The first noise control module based on the FCN network serves as the input. With the second noise control module Every 20 steps ( Randomly sampled Gaussian noise Calculate the actual noise :: , Among them, based on The first modulation parameter is calculated and used for... Scaling, based on The calculated second modulation parameter is used for... Offset using actual noise Add noise.

[0029] According to the diffusion model principle, the time step The noise addition formula is: , in, , These are hyperparameters, determined based on the degree of reconstruction in the graph-to-graph transformation. The greater the degree of reconstruction, the larger these two hyperparameters should be, along with the time step. The maximum value of 1000 was set manually; among them, , , , in, Indicates time step Noise control parameters at that time, Represents the cumulative noise figure. Indicates time step The noise variance coefficient at that time, and They represent the maximum noise variance, respectively. and minimum noise variance , Represents the total number of time steps , Take 80, Take 200, during training The values ​​are selected while retaining a certain degree of randomness. It gradually decreases within the range.

[0030] S3 freezes the parameters of the pre-trained diffusion model and constructs a loss function based on the predicted latent variables and the target latent variables, which is used to train the two noise control modules.

[0031] In the embodiment, the predicted latent variables are obtained based on the final output of the diffusion model. Its corresponding target latent variable A mean squared error loss is constructed between the two noise control modules, and the first and second noise control modules are trained using error backpropagation.

[0032] Specifically, based on the Gaussian noise that is sampled every 20 steps as mentioned above... That is: maintain the Gaussian noise in step S2. Keeping the error constant, perform 20 backpropagations to train the noise control module. and Then replace the Gaussian noise used. The initial learning rate is 0.0001, and the learning rate is multiplied by 0.95 every 400 steps. The training takes 20,000 steps (each backpropagation of error is considered as one step), and finally two noise control modules are obtained.

[0033] S4: Freeze the parameters of the two trained noise control modules, construct a loss function based on the actual predicted noise and the target predicted noise of the diffusion model, and use it to fine-tune the pre-trained diffusion model.

[0034] In this embodiment, the pre-trained noise control module is used to fine-tune the pre-trained U-Net-based diffusion model to obtain a new U-Net.

[0035] Specifically, freeze the parameters of all modules except U-Net, and denote U-Net as... ,in The network parameters are represented as follows: the actual predicted noise during the denoising process is represented as... Target latent variables Introduce the noise to be predicted, and record the target predicted noise. for: .

[0036] The loss function for the fine-tuning process is expressed as: , in, Indicates the mean squared error. To utilize the noisy latent variable obtained by adding noise using the noise control module, the time step of adding noise is... And it is randomly generated, and can be combined with The same can also be a subset or a similar range. The learning rate is 0.00002, and the training takes 16,000 steps to finally obtain the new U-Net.

[0037] S5. For a new input image, two trained noise control modules are used to add noise to the image, and then a finely tuned diffusion model is used to remove noise, restoring the image to the image space to obtain the generated image.

[0038] In this embodiment, for a new image to be processed (obtained by VAE compression), a new actual noise is obtained using the noise control module, and hyperparameters are obtained using this actual noise. After adding noise as a time step, the finely tuned U-Net is used as a noise predictor for denoising. The image is then restored to the image space using VAE to obtain the output image.

[0039] Specifically, a new mural image is entered (such as...) Figure 3As shown in (a) above, the value can be truncated to 512*512 or scaled down to 512*512. Using VAE, it can be compressed to 4*64*64 to obtain a new latent variable. Using a noise control module and get ( (This is Gaussian noise), and this noise is used to obtain... After adding noise at specific time steps, DDIM denoising is performed using a finely tuned U-Net (iteration time steps are 100, 90, 80, ..., 0). The image is then restored to its original image space using a VAE to obtain the final restored output mural image (e.g., ...). Figure 3 As shown in (b), stains and damage in the mural were effectively removed without affecting the outline and structure.

[0040] Example 2 This embodiment provides a graph-to-graph transformation method that utilizes a noise control module and diffusion model principles for fine-tuning, applied to style transfer. The similarities to Embodiment 1 will not be repeated here, and the method includes the following steps: S1, compress the original image and the target image into the latent space to obtain the original latent variables and the target latent variables.

[0041] The original images consist of 100 real images, and the target images consist of 100 corresponding cartoon-style images, each with a size of 512*512 pixels (RGB 3 channels).

[0042] S2, the original latent variables are input into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate actual noise. The original latent variables are then denoised using the actual noise to obtain noisy latent variables. The noisy latent variables are then denoised using a pre-trained diffusion model to obtain the predicted latent variables.

[0043] Among them, time step Scope Pick During the training process The value gradually decreases within this range while retaining a certain degree of randomness.

[0044] S3 freezes the parameters of the pre-trained diffusion model and constructs a loss function based on the predicted latent variables and the target latent variables, which is used to train the two noise control modules.

[0045] Here, the initial learning rate is 0.00005, and the learning rate is multiplied by 0.98 every 200 steps, with a learning step count of 2000.

[0046] S4: Freeze the parameters of the two trained noise control modules, construct a loss function based on the actual predicted noise and the target predicted noise of the diffusion model, and use it to fine-tune the pre-trained diffusion model.

[0047] The learning rate was 0.00002, and the number of training steps was 8000.

[0048] For a new input image, S5 uses two trained noise control modules to add noise, then uses a fine-tuned diffusion model to remove noise, restoring the image to the image space to obtain the generated image.

[0049] The time step is 350, and the input is a 512*512 real bird image (e.g., Figure 4 As shown in (a) above, you can obtain cartoon-style bird images of the same size with the same key parts and features (such as...). Figure 4 (as shown in (b)).

[0050] Based on the same inventive concept, such as Figure 5 As shown, this embodiment of the invention also provides an image-to-image conversion device that uses a noise control module and a diffusion model principle for fine-tuning, including: an image compression module 510, a noise modulation module 520, a noise training module 530, a noise reduction fine-tuning module 540, and an image generation module 550.

[0051] The image compression module 510 is used to compress the original image and the target image into the latent space to obtain the original latent variables and the target latent variables.

[0052] The noise modulation module 520 is used to input the original latent variables into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter. These parameters are used to scale and offset the standard Gaussian noise to generate actual noise. The original latent variables are then denoised using the actual noise to obtain noisy latent variables. Finally, the noisy latent variables are denoised using a pre-trained diffusion model to obtain the predicted latent variables.

[0053] The noise-adding training module 530 is used to freeze the parameters of the pre-trained diffusion model and construct a loss function based on the predicted latent variables and the target latent variables for training the two noise control modules.

[0054] The denoising fine-tuning module 540 is used to freeze the parameters of the two trained noise control modules, construct a loss function based on the actual predicted noise and the target predicted noise of the diffusion model, and use it to fine-tune the pre-trained diffusion model.

[0055] The image generation module 550 is used to partially add noise to a new input image using two trained noise control modules, and then use a fine-tuned diffusion model to remove noise, restoring the image to the image space to obtain the generated image.

[0056] It should be noted that the image-to-image conversion device for fine-tuning using the noise control module and diffusion model principle provided in the above embodiments belongs to the same inventive concept as the image-to-image conversion method for fine-tuning using the noise control module and diffusion model principle. For details of its implementation process, please refer to the embodiment of the image-to-image conversion method for fine-tuning using the noise control module and diffusion model principle, which will not be repeated here.

[0057] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A graph-to-graph conversion method for fine-tuning using a noise control module and a diffusion model principle, characterized in that, Includes the following steps: The original image and the target image are compressed into the latent space to obtain the original latent variables and the target latent variables; The original latent variables are input into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate the actual noise. The original latent variables are then denoised using the actual noise to obtain the noisy latent variables. The noisy latent variables are then denoised using a pre-trained diffusion model to obtain the predicted latent variables. The parameters of the pre-trained diffusion model are frozen, and a loss function is constructed based on the predicted latent variables and the target latent variables to train the two noise control modules. The parameters of the two pre-trained noise control modules are frozen. A loss function is constructed based on the actual predicted noise and the target predicted noise of the diffusion model. This function is used to fine-tune the pre-trained diffusion model, including: the diffusion model. The actual predicted noise in the denoising process is represented as: ,in Represents network parameters, Indicates the original latent variables The noisy latent variable is obtained by adding noise through two noise control modules; target prediction noise. Represented as: ,in, Indicates time step The cumulative noise figure at that time The target latent variable is represented; the loss function for the fine-tuning process is expressed as: ,in, Indicates the mean squared error; For a new input image, two trained noise control modules are used to add noise to the image, and then a finely tuned diffusion model is used to remove noise, restoring the image to the image space to obtain the generated image.

2. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 1, characterized in that, The step of inputting the original latent variables into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate the actual noise, includes: Using two original latent variables The first noise control module as input With the second noise control module Sampling standard Gaussian noise Calculate the actual noise : ， Among them, based on The first modulation parameter is calculated and used for... Scaling, based on The calculated second modulation parameter is used for... Perform an offset.

3. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 2, characterized in that, The method of adding noise to the original latent variables using actual noise to obtain noisy latent variables includes: Time step The noise addition formula is: ， in, Indicates noisy latent variables; ， ， ， in, Indicates time step Noise control parameters at that time, Represents the cumulative noise figure. Indicates time step The noise variance coefficient at that time, and These represent the maximum noise variance and the minimum noise variance, respectively. This represents the total number of time steps.

4. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 1 or 2, characterized in that, When generating actual noise, each The standard Gaussian noise of the sample is replaced step by step to construct a new distribution of the actual noise.

5. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 1, characterized in that, Both the first noise control module and the second noise control module are built based on fully convolutional networks.

6. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 1, characterized in that, The diffusion model is built based on the U-Net network.

7. The graph-to-graph conversion method for fine-tuning using a noise control module and diffusion model principle as described in claim 1, characterized in that, A variational autoencoder is used for image compression and image restoration.

8. A graph-to-graph conversion device for fine-tuning using a noise control module and a diffusion model principle, implemented based on the graph-to-graph conversion method for fine-tuning using a noise control module and a diffusion model principle as described in any one of claims 1 to 7, characterized in that, include: The system includes an image compression module, a noise modulation module, a noise addition training module, a noise reduction fine-tuning module, and an image generation module. The image compression module is used to compress the original image and the target image into the latent space to obtain the original latent variables and the target latent variables; The noise modulation module is used to input the original latent variables into the first noise control module and the second noise control module respectively to obtain the first modulation parameter and the second modulation parameter, which are used to scale and offset the standard Gaussian noise to generate actual noise. The original latent variables are then denoised using the actual noise to obtain noisy latent variables. Finally, the noisy latent variables are denoised using a pre-trained diffusion model to obtain the predicted latent variables. The noise-adding training module is used to freeze the parameters of the pre-trained diffusion model and construct a loss function based on the predicted latent variables and the target latent variables, which is used to train the two noise control modules. The denoising fine-tuning module is used to freeze the parameters of the two trained noise control modules, and construct a loss function based on the actual predicted noise and target predicted noise of the diffusion model, which is used to fine-tune the pre-trained diffusion model. The image generation module is used to partially add noise to a new input image using two trained noise control modules, and then use a fine-tuned diffusion model to remove noise, restoring the image to the image space to obtain the generated image.

9. A method for mural restoration or style transfer, characterized in that, Based on the image-to-image conversion method described in any one of claims 1 to 7, which utilizes a noise control module and diffusion model principle for fine-tuning, this method is used to restore the damage to the mural or to generate images for image style changes.

Images