Image restoration method and device based on diffusion model and generative adversarial training

Abstract

The embodiment of the invention provides an image restoration method and device based on a diffusion model and generative adversarial training, and the method comprises the steps: obtaining image data, inputting the image data into a degeneration encoder, and obtaining a first result; providing an interaction page, and obtaining image limitation information based on the interaction page; the first result and the image limiting information are input into an image restoration network to obtain a second result, and the image restoration network is generated based on a diffusion model in a pre-trained diffusion network; inputting the second result into an image decoder for processing to obtain a restoration result, and feeding back the restoration result; according to the scheme, the pre-trained encoder and the diffusion model can be finely adjusted so as to be suitable for restoration of the degraded image, and a more accurate restored image can be obtained.

Description

Technical Field

[0001] The present application relates to the field of computer information technology. Specifically, the present application relates to an image restoration method and device based on a diffusion model and generative adversarial training. Background Art

[0002] From the perspective of deep learning, image restoration models aim to learn the mapping from the degraded image space to the natural image space.

[0003] Existing solutions usually use adversarial training to train image restoration models, then input the image into the trained model and restore the image through the model. However, using this solution, the restored image effect is poor. Summary of the Invention

[0004] The embodiments of the present application provide an image restoration method and device based on a diffusion model and generative adversarial training, which can be fine-tuned based on a pre-trained diffusion network to obtain a fine-tuned encoder, image restoration network, etc., which are suitable for processing degraded images and improving image restoration effects.

[0005] The technical solution is as follows:

[0006] In a first aspect, the present application provides an image restoration method based on a diffusion model and generative adversarial training, the method comprising: acquiring image data, and inputting the image data into a degradation removal encoder to obtain a first result; providing an interactive page, and acquiring image limitation information based on the interactive page; inputting the first result and the image limitation information into an image restoration network to obtain a second result, wherein the image restoration network is generated based on a diffusion model in a pre-trained diffusion network; inputting the second result into an image decoder for processing to obtain a restoration result, and feeding it back.

[0007] Furthermore, the pre-trained diffusion network includes a variational autoencoder; the degradation removal encoder is generated by the following steps: obtaining the variational autoencoder and first training data in the pre-trained diffusion network, the first training data including image data corresponding to each degradation mode, the degradation mode including blur, noise, missing, compression, etc.; using the first training data to fine-tune the variational autoencoder to obtain the degradation removal encoder.

[0008] Furthermore, the image restoration network is generated by the following steps: obtaining a diffusion model in a pre-trained diffusion network and initializing it to obtain a generator; obtaining second training data and adopting a low-rank adaptation method to fine-tune the generator based on the second training data to obtain an image restoration network.

[0009] Furthermore, the method of adopting low-rank adaptation fine-tunes the generator based on the second training data to obtain an image restoration network, including: obtaining first parameter information of the generator and determining a low-rank decomposition matrix; training the generator based on the second training data, and adjusting the low-rank decomposition matrix to determine second parameter information, so as to form an image restoration network based on the first parameter information and the second parameter information.

[0010] Furthermore, the generator is fine-tuned based on the second training data to obtain an image restoration network, including: generating a first adversarial discriminator; inputting the second training data into the generator and the image decoder to obtain a first intermediate result; discriminating the first intermediate result based on the first adversarial discriminator to obtain a first discrimination result; and performing adversarial training based on the first intermediate result and the first discrimination result to obtain an image restoration network.

[0011] Furthermore, the obtaining of image limitation information based on the interactive page includes: obtaining text prompt information based on the first interactive information with the interactive page; obtaining generation force information based on the second interactive information with the interactive page; and obtaining texture sharpness information based on the third interactive information with the interactive page.

[0012] In a second aspect, the present application provides an image restoration device based on a diffusion model and generative adversarial training, the method comprising: an image data acquisition module for acquiring image data and inputting the image data into a degradation removal encoder to obtain a first result; a limitation information acquisition module for providing an interactive page and acquiring image limitation information based on the interactive page; an image restoration processing module for inputting the first result and the image limitation information into an image restoration network to obtain a second result, wherein the image restoration network is generated based on a diffusion model in a pre-trained diffusion network; and a restoration result acquisition module for inputting the second result into an image decoder for processing to obtain a restoration result and provide feedback.

[0013] In a third aspect, the present application provides a network device comprising: a memory, a transceiver, and a processor; wherein the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor; and the processor is used to read the computer program in the memory and execute the method described in the first aspect.

[0014] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, which implements the method described in the first aspect when the computer program is executed by a processor.

[0015] The beneficial effects of the technical solution provided by this application are:

[0016] The solution of the present application can be applied in the scenario of image data restoration. It can fine-tune the pre-trained diffusion network for blurred images, noisy images, and images with missing data to obtain an encoder, image restoration network, and decoder suitable for degraded images. Moreover, the solution can perform targeted image restoration based on user prompts to improve image effects. Specifically, the solution uses the variational autoencoder, diffusion model, and decoder in the pre-trained diffusion network for fine-tuning to obtain a de-degradation encoder, image restoration network, and decoder suitable for image restoration scenarios. Furthermore, the image restoration network can also obtain user prompts and perform targeted image restoration based on the user prompts. The solution can obtain image data to be restored and input the image data into the de-degradation encoder to obtain a first result; and provide an interactive page to obtain image limitation information based on the interactive page; input the first result and the image limitation information into the image restoration network to obtain a second result; input the second result into the image decoder for processing to obtain a restoration result, and provide feedback. This solution uses a pre-trained diffusion network to fine-tune the model to repair the image, which can reduce the training process. It can also use low-rank adaptation to fine-tune the model, thereby reducing the amount of data that needs to be stored. It also uses adversarial training to reduce the entire training iteration process, making the model converge more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0018] Figure 1 This is a schematic diagram of the overall process of an image restoration method based on a diffusion model and generative adversarial training according to an embodiment of the present application;

[0019] Figure 2 1 is a schematic diagram of an encoder and a diffusion model fine-tuning process of an image restoration method based on a diffusion model and generative adversarial training according to an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the steps of an image restoration method based on a diffusion model and generative adversarial training according to an embodiment of the present application;

[0021] Figure 4 1 is a structural diagram of a feature-based mask extraction and alignment device according to an embodiment of the present application;

[0022] Figure 5 This is a structural block diagram of a network device according to an embodiment of the present application;

[0023] Figure 6 This is a structural block diagram of a user device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout identify the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present application and are not to be construed as limiting the present application.

[0025] Those skilled in the art will understand that, unless otherwise specified, the singular forms "a," "an," "the," and "the" used herein may also include the plural forms, and "a plurality" refers to two or more, and other quantifiers are similar. It should be further understood that the term "comprising" used in the specification of this application refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling. The term "and / or" used herein describes the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0026] The scheme of the present application can be applied in the scene of image data restoration, can be based on the pre-trained diffusion network for fine-tuning for blurred images, noisy images, images with missing data, obtain the encoder suitable for the degraded image, image restoration network, decoder; and the scheme can be based on the user's prompt, targeted image restoration, improve the image effect; specifically, the scheme uses the variational autoencoder in the pre-trained diffusion network, diffusion model, decoder for fine-tuning processing, obtains the removal of the degraded encoder, image repair network and decoder suitable for the image restoration scene, and the image repair network can also obtain the user's prompt, and according to the user's prompt, targeted image restoration is performed. The present scheme can obtain the image data to be restored, and input the image data into the removal of the degraded encoder to obtain a first result; and provide an interactive page, obtain image limiting information based on the interactive page; input the first result and the image limiting information into the image repair network to obtain a second result; input the second result into the image decoder for processing to obtain a repair result, and feedback. The present scheme uses the pre-trained diffusion network to fine-tune the model of the repaired image, which can reduce the training process, and can use the low-rank adaptive method to fine-tune the model, thereby reducing the amount of data to be stored, and also uses the way of adversarial training to reduce the iteration process of the whole training, so that the model converges more quickly.

[0027] From the perspective of deep learning, image restoration models aim to learn the mapping from the degraded image space to the natural image space. However, they face several key challenges: removing degradation, generating realistic details, and ensuring pixel-level consistency. Over time, three major categories of methods have emerged: mean square error (MSE) based, generative adversarial network (GAN) based, and diffusion model based. However, they have yet to achieve a good balance between restoration quality, fidelity, and speed.

[0028] The present scheme proposes a method of image repair (HYPIR) to address these challenges. In the present scheme: first, use a diffusion model for initialization, then fine-tune through lightweight adversarial training without relying on diffusion loss, iterative sampling, or additional adapters. The present scheme analyzes the similarity between the posterior distribution of degraded samples and the noise samples in the diffusion model, and constructs a reasonable starting point for the restoration model, which can significantly improve the convergence speed of adversarial training. In addition, HYPIR inherits the ability of diffusion models, with rich user control, supporting text-guided restoration and adjustable texture richness. With only one forward propagation, it can achieve faster convergence and inference speed than diffusion-based methods.

[0029] Extensive experiments demonstrate that HYPIR consistently outperforms previous state-of-the-art methods, establishing a new paradigm for efficient, high-quality, and high-resolution image restoration. In summary, HYPIR combines a concise methodology, comprehensive demonstration, and excellent restoration performance.

[0030] The following is an explanation of some key words in this plan:

[0031] Image restoration is an important task in computer vision, aiming to recover clear visual information from degraded or damaged images. Common types of degradation include noise, blur, and missing data. Image restoration technology is widely used in fields such as medical imaging, satellite image processing, and digital image inpainting. Through advanced methods such as deep learning, researchers can effectively improve the quality of restoration, generate more realistic details, and maintain pixel-level consistency. In recent years, methods based on generative adversarial networks (GANs) and diffusion models have made significant progress in image restoration.

[0032] Adversarial training is the core mechanism of generative adversarial networks (GANs), involving two neural networks: the generator and the discriminator. The generator's task is to generate realistic samples to confuse the discriminator, while the discriminator's task is to distinguish between real and generated samples. Through this adversarial process, the generator continuously improves its generation capabilities, striving to produce more realistic images, while the discriminator improves its discrimination capabilities. Ultimately, ideally, the samples generated by the generator will be difficult for the discriminator to identify as fake. This process forces the two networks to mutually optimize in a game of chess, achieving higher generation quality.

[0033] Diffusion models are an emerging generative model that generates samples by simulating a gradual diffusion process from real data to noise. The model first gradually adds noise to the data to form a noise distribution, and then gradually removes the noise through a reverse process to restore the original data. Diffusion models have demonstrated excellent performance in tasks such as image generation, image restoration, and super-resolution, generating high-quality and diverse images. Compared to generative adversarial networks (GANs), diffusion models offer significant advantages in training stability and generation quality, and have attracted widespread attention in the field of generative models in recent years.

[0034] In this work, we introduce a novel methodology for building image restoration models specifically designed to address these three key challenges. Our approach is straightforward: efficient parameter initialization using a pre-trained diffusion model, followed by lightweight adversarial fine-tuning via LoRA to specifically adapt it to the image restoration task. Notably, our approach does not rely on diffusion teachers, ControlNet adapters, iterative refinement steps, or multi-stage training—essential components of previous approaches. As a result, training and inference speeds are an order of magnitude faster than diffusion-based methods, while achieving superior performance! By cleverly combining the strengths of existing paradigms, our framework establishes a new research paradigm. We name our proposed method HYPIR.

[0035] In comparison, existing solutions are based on GANs or diffusion models. This solution combines diffusion and GANs. Among existing solutions, traditional GAN-based methods are fast, but training is unstable and results are poor. Diffusion-based methods are effective, but inference speed is slow. This solution combines excellent restoration results with inference speed, achieving restoration results comparable to or even exceeding diffusion methods, while approaching the speed of GANs. Among existing solutions, the maximum number of model parameters is 2 billion. This solution increases the number of model parameters to 12 billion.

[0036] The following is an introduction to the processing flow of this solution: Figure 1 and 2 As shown in Figure 1, the first step of this solution is to address the complexity of image restoration tasks due to the diverse types of image degradation in the real world, including blur, noise, and missing data. To effectively address these diverse degradation scenarios, the encoder in the variational autoencoder (VAE) is fine-tuned. This fine-tuning of the encoder allows it to better adapt to various degradation characteristics, thereby improving its robustness and accuracy in removing degradation.

[0037] The second step is to initialize the generator using the pre-trained diffusion model parameters. This process can leverage the rich features and semantic information learned in the pre-trained model, providing a good starting point for the generator. Next, a method based on low-rank adaptation (LoRA) is used to fine-tune the generator. LoRA is an efficient adaptation technique that adjusts model parameters by introducing a low-rank matrix, thereby achieving more flexible learning and adaptability.

[0038] During fine-tuning, an adversarial training strategy is incorporated to further improve the generator's performance. Adversarial training introduces a discriminator to compete with the generator, causing it to focus more on the features and details of real images when generating images, thereby improving the realism and quality of the generated images. This combination can effectively enhance the robustness of the generator across a variety of tasks, enabling it to produce more realistic and detailed images. Therefore, the core framework of the method consists of initializing the generator using a pre-trained diffusion model and fine-tuning it using LoRA and adversarial training strategies. This approach not only improves the efficiency of the generator but also significantly enhances its performance in complex image generation tasks.

[0039] The following is an introduction to the encoder, repair network and other processing processes of this solution:

[0040] Degradation removal encoder: Traditional image encoders can only process clean images and cannot handle different degradation patterns (noise, blur, compression, etc.). This solution innovatively retrains the encoder and fine-tunes it into a degradation removal encoder that is robust to different degradation patterns and can pre-process different low-quality images.

[0041] Image Inpainting Network: Traditional diffusion network-based image inpainting methods consider the text-based graph model and the image inpainting model to be two different tasks and cannot be directly used. Therefore, an additional network control module (such as ControlNet) is required in the structure. This approach significantly increases the number of network parameters and the amount of training computation. This solution innovatively directly replicates the diffusion network structure as the inpainting network structure, eliminating the need for an additional control module and adding no additional computation.

[0042] Parameter initialization: Traditional GAN-based image restoration algorithms have fast inference speeds, but unstable training and poor results. Traditional diffusion-based methods are effective, but slow inference speeds and require additional control modules for retraining. This solution innovatively unifies the diffusion and restoration models. Initialization directly uses the diffusion network parameters, eliminating the need for complex control modules and the large number of parameters and computational overhead. Using LORA to fine-tune the model, training converges rapidly.

[0043] User interaction: Traditional methods can accept user instructions, but the results are not satisfactory. This solution model has excellent user interaction and can personalize the output according to the user's selection.

[0044] GAN training: Using GANs to train an image inpainting network initialized with a diffusion model, we found rapid convergence. During testing, the network only required one forward propagation step. This leverages the excellent performance of the diffusion model while also achieving the fast inference capabilities of GANs.

[0045] This paper proposes a novel and extremely concise image restoration framework, HYPIR. By leveraging the strong priors contained in a pre-trained diffusion model for initialization and employing lightweight adversarial fine-tuning based on LoRA, it achieves fast convergence, numerical stability, and excellent restoration quality. Unlike traditional diffusion-based methods, HYPIR completely eliminates iterative sampling, distillation complexity, and computationally intensive auxiliary networks. As a result, it significantly accelerates training and inference without sacrificing restoration performance. In fact, it consistently outperforms state-of-the-art methods. Furthermore, HYPIR uniquely empowers users with intuitive control mechanisms such as textual hints, adjustable texture richness, and customizable fidelity.

[0046] Specifically, this application provides an image restoration method based on diffusion model and generative adversarial training, such as Figure 3 As shown, the method includes:

[0047] Step 102: Acquire image data, and input the image data into a de-degradation encoder to obtain a first result.

[0048] Step 104: Provide an interactive page, and obtain image limitation information based on the interactive page.

[0049] Step 106: Input the first result and the image limitation information into an image restoration network to obtain a second result, wherein the image restoration network is generated based on a diffusion model in a pre-trained diffusion network.

[0050] Step 108: Input the second result into the image decoder for processing to obtain a restoration result, and feed it back.

[0051] The solution of the present application can be applied in the scenario of image data restoration. It can fine-tune the pre-trained diffusion network for blurred images, noisy images, and images with missing data to obtain an encoder, image restoration network, and decoder suitable for degraded images. Moreover, the solution can perform targeted image restoration based on user prompts to improve image effects. Specifically, the solution uses the variational autoencoder, diffusion model, and decoder in the pre-trained diffusion network for fine-tuning to obtain a de-degradation encoder, image restoration network, and decoder suitable for image restoration scenarios. Furthermore, the image restoration network can also obtain user prompts and perform targeted image restoration based on the user prompts. The solution can obtain image data to be restored and input the image data into the de-degradation encoder to obtain a first result; and provide an interactive page to obtain image limitation information based on the interactive page; input the first result and the image limitation information into the image restoration network to obtain a second result; input the second result into the image decoder for processing to obtain a restoration result, and provide feedback. This solution uses a pre-trained diffusion network to fine-tune the model to repair the image, which can reduce the training process. It can also use low-rank adaptation to fine-tune the model, thereby reducing the amount of data that needs to be stored. It also uses adversarial training to reduce the entire training iteration process, making the model converge more quickly.

[0052] This solution can be based on the encoder of a variational autoencoder in a pretrained diffusion network, and can be trained and fine-tuned using data corresponding to degraded images to obtain a degradation-removing encoder suitable for different types of degradation. Specifically, as an optional embodiment, the pretrained diffusion network includes a variational autoencoder; the degradation-removing encoder is generated by the following steps: obtaining the variational autoencoder in the pretrained diffusion network and first training data, the first training data including image data corresponding to various degradation modes, including blur, noise, missing, and compression; and fine-tuning the variational autoencoder using the first training data to obtain the degradation-removing encoder. The degradation-removing encoder can extract image features.

[0053] This solution can be initialized using a pre-trained diffusion model and fine-tuned using data to obtain an image restoration network. Specifically, as an optional embodiment, the image restoration network is generated by the following steps: obtaining the diffusion model in the pre-trained diffusion network and initializing it to obtain a generator; obtaining second training data and fine-tuning the generator based on the second training data using a low-rank adaptation method to obtain an image restoration network. The image restoration network can be processed based on the features extracted by the encoder to restore the image. The image restoration network can complete image restoration by adding noise and removing noise. The training data corresponding to the image restoration network and the data of the degraded encoder can be the same set of data or different data.

[0054] This solution can adopt a low-rank adaptation method to fine-tune the generator, thereby reducing the amount of parameter data that needs to be stored. Specifically, as an optional embodiment, the method of adopting low-rank adaptation fine-tunes the generator based on the second training data to obtain an image restoration network, including: obtaining first parameter information of the generator and determining a low-rank decomposition matrix; training the generator based on the second training data and adjusting the low-rank decomposition matrix to determine second parameter information, so as to form an image restoration network based on the first parameter information and the second parameter information.

[0055] For the final output result, this solution can be processed based on adversarial training. Specifically, as an optional embodiment, the generator is fine-tuned based on the second training data to obtain an image restoration network, including: generating a first adversarial discriminator; inputting the second training data into the generator and the image decoder to obtain a first intermediate result; discriminating the first intermediate result based on the first adversarial discriminator to obtain a first discrimination result; and performing adversarial training based on the first intermediate result and the first discrimination result to obtain an image restoration network.

[0056] For the image encoder, this solution can directly utilize the decoder in a pretrained diffusion network for initialization. Specifically, as an optional embodiment, the pretrained diffusion network includes a decoder, and the method further includes obtaining and initializing the decoder of the pretrained diffusion network to obtain an image decoder. The image encoder can encode based on the features to obtain a restored image for output to the user.

[0057] This solution can set up an interactive page, and the interactive page can be equipped with a corresponding plug-in to interact with the user to obtain information such as text prompts, texture sharpness, and build strength, and output limited information to the image restoration network for targeted optimization processing. Specifically, as an optional embodiment, obtaining image limited information based on the interactive page includes: obtaining text prompt information based on a first interaction with the interactive page; obtaining build strength information based on a second interaction with the interactive page; and obtaining texture sharpness information based on a third interaction with the interactive page.

[0058] On the basis of the above embodiments, the present application also provides an image restoration device based on a diffusion model and generative adversarial training, such as Figure 4 As shown, the method includes:

[0059] The image data acquisition module 202 is used to acquire image data and input the image data into the degradation removal encoder to obtain a first result.

[0060] The limitation information acquisition module 204 is used to provide an interactive page and acquire image limitation information based on the interactive page.

[0061] The image restoration processing module 206 is configured to input the first result and the image limitation information into an image restoration network to obtain a second result. The image restoration network is generated based on a diffusion model in a pre-trained diffusion network.

[0062] The restoration result acquisition module 208 is configured to input the second result into the image decoder for processing, obtain a restoration result, and provide feedback.

[0063] The implementation of the embodiment of the present application is similar to that of the above embodiment. The specific implementation can refer to the specific implementation of the above embodiment and will not be repeated here.

[0064] The solution of the present application can be applied in the scenario of image data restoration. It can fine-tune the pre-trained diffusion network for blurred images, noisy images, and images with missing data to obtain an encoder, image restoration network, and decoder suitable for degraded images. Moreover, the solution can perform targeted image restoration based on user prompts to improve image effects. Specifically, the solution uses the variational autoencoder, diffusion model, and decoder in the pre-trained diffusion network for fine-tuning to obtain a de-degradation encoder, image restoration network, and decoder suitable for image restoration scenarios. Furthermore, the image restoration network can also obtain user prompts and perform targeted image restoration based on the user prompts. The solution can obtain image data to be restored and input the image data into the de-degradation encoder to obtain a first result; and provide an interactive page to obtain image limitation information based on the interactive page; input the first result and the image limitation information into the image restoration network to obtain a second result; input the second result into the image decoder for processing to obtain a restoration result, and provide feedback. This solution uses a pre-trained diffusion network to fine-tune the model to repair the image, which can reduce the training process. It can also use low-rank adaptation to fine-tune the model, thereby reducing the amount of data that needs to be stored. It also uses adversarial training to reduce the entire training iteration process, making the model converge more quickly.

[0065] It should be noted that the division of units and / or modules in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, there may be other division methods. In addition, the functional units and / or modules in the various embodiments of the present application may be integrated into one processing unit and / or module, or each unit and / or module may exist physically alone, or two or more units and / or modules may be integrated into one unit and / or module. The above-mentioned integrated units and / or modules may be implemented in the form of hardware or in the form of software functional units and / or modules.

[0066] If the integrated units and / or modules are implemented in the form of software functional units and / or modules and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0067] In addition, the data transmission device and data transmission method provided in the above embodiments are based on the same application concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0068] Figure 5 A structural block diagram of a network device is shown according to an exemplary embodiment.

[0069] like Figure 5 As shown, the network device 1100 includes at least: a processor 1110 , a memory 1120 and a transceiver 1130 .

[0070] The transceiver 1130 is used to receive and send data under the control of the processor 1110 .

[0071] exist Figure 5 In the embodiment of the present invention, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1110 and memory represented by memory 1120. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and therefore will not be further described herein. The bus interface provides an interface. The transceiver 1130 may be a plurality of components, i.e., a transmitter and a receiver, providing units and / or modules for communicating with various other devices over a transmission medium, such as a wireless channel, a wired channel, an optical cable, or the like.

[0072] The processor 1110 is responsible for managing the bus architecture and general processing, and the memory 1120 can store data used by the processor 1110 when performing operations.

[0073] Optionally, the processor 1110 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor 1110 may also employ a multi-core architecture. The processor 1110 and the memory 1120 may also be physically separated.

[0074] The processor 1110 calls the computer program stored in the memory 1120 to execute any one of the methods for allocating a cell radio network temporary identifier provided in the above embodiments of the present application according to the obtained executable instructions.

[0075] Figure 6 A structural block diagram of a user equipment is shown according to an exemplary embodiment.

[0076] like Figure 6 As shown, the user equipment 1300 includes at least: a processor 1310 , a memory 1320 and a transceiver 1330 .

[0077] The transceiver 1330 is used to receive and send data under the control of the processor 1310.

[0078] exist Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 1310 and memory represented by memory 1320. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1330 may be a plurality of components, i.e., a transmitter and a receiver, providing units and / or modules for communicating with various other devices on a transmission medium, such as wireless channels, wired channels, optical cables, and other transmission media. For different user devices, the user interface 1340 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0079] The processor 1310 is responsible for managing the bus architecture and general processing, and the memory 1320 can store data used by the processor 1310 when performing operations.

[0080] Optionally, the processor 1310 may be a CPU (central processing unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device). The processor 1310 may also employ a multi-core architecture. The processor 1310 and the memory 1320 may also be physically separated.

[0081] The processor 1310 calls the computer program stored in the memory 1320 to execute any one of the methods for allocating a cell radio network temporary identifier provided in the above embodiments of the present application according to the obtained executable instructions.

[0082] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0083] In addition, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the data transmission method of each of the above embodiments. The storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as a floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0084] In an embodiment of the present application, a program product is provided. For example, the program product is an FPGA chip or a DSP chip. The program product includes executable instructions stored in a storage medium. A processor reads the executable instructions from the storage medium, so that when the executable instructions are executed by the processor, the data transmission method described in each of the above embodiments is implemented.

[0085] The solution of the present application can be applied in the scenario of image data restoration. It can fine-tune the pre-trained diffusion network for blurred images, noisy images, and images with missing data to obtain an encoder, image restoration network, and decoder suitable for degraded images. Moreover, the solution can perform targeted image restoration based on user prompts to improve image effects. Specifically, the solution uses the variational autoencoder, diffusion model, and decoder in the pre-trained diffusion network for fine-tuning to obtain a de-degradation encoder, image restoration network, and decoder suitable for image restoration scenarios. Furthermore, the image restoration network can also obtain user prompts and perform targeted image restoration based on the user prompts. The solution can obtain image data to be restored and input the image data into the de-degradation encoder to obtain a first result; and provide an interactive page to obtain image limitation information based on the interactive page; input the first result and the image limitation information into the image restoration network to obtain a second result; input the second result into the image decoder for processing to obtain a restoration result, and provide feedback. This solution uses a pre-trained diffusion network to fine-tune the model to repair the image, which can reduce the training process. It can also use low-rank adaptation to fine-tune the model, thereby reducing the amount of data that needs to be stored. It also uses adversarial training to reduce the entire training iteration process, making the model converge more quickly.

[0086] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) that contain computer-usable program code.

[0087] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor-readable memory produce an article of manufacture comprising an instruction device that implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These processor-executable instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0090] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0091] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be included in the scope of protection of the present application.

Claims

1. An image restoration method based on diffusion model and generative adversarial training, characterized in that: The method comprises: Acquire image data, and input the image data into a degradation removal encoder to obtain a first result; Providing an interactive page and obtaining image-limited information based on the interactive page; Inputting the first result and image qualification information into an image restoration network to obtain a second result, wherein the image restoration network is generated based on a diffusion model in a pre-trained diffusion network; The second result is input into the image decoder for processing to obtain a restoration result, which is fed back.

2. The method according to claim 1, wherein The pre-trained diffusion network includes a variational autoencoder; the degradation removal encoder is generated by the following steps: Obtaining a variational autoencoder and first training data in a pre-trained diffusion network, where the first training data includes image data corresponding to each degradation mode, where the degradation mode includes blur, noise, missing, and compression; The first training data is used to fine-tune the variational autoencoder to obtain a degradation-removed encoder.

3. The method according to claim 1, wherein The image restoration network is generated by the following steps: Obtain the diffusion model in the pre-trained diffusion network and initialize it to obtain the generator; The second training data is obtained, and a low-rank adaptation method is adopted to fine-tune the generator based on the second training data to obtain an image restoration network.

4. The method according to claim 3, wherein The low-rank adaptation method is used to fine-tune the generator based on the second training data to obtain an image restoration network, including: Obtain the first parameter information of the generator and determine the low-rank decomposition matrix; The generator is trained based on the second training data, and the low-rank decomposition matrix is ​​adjusted to determine second parameter information to form an image restoration network based on the first parameter information and the second parameter information.

5. The method according to claim 3, wherein Fine-tuning the generator based on the second training data to obtain an image restoration network includes: Generate the first adversarial discriminator; Inputting the second training data into the generator and the image decoder to obtain a first intermediate result; Discriminate the first intermediate result based on the first adversarial discriminator to obtain a first discrimination result; Adversarial training is performed based on the first intermediate result and the first discrimination result to obtain an image restoration network.

6. The method according to claim 1, wherein The acquiring of image limitation information based on the interactive page includes: Acquire text prompt information based on first interaction information with the interactive page; Generating strength information based on second interaction information with the interaction page; Texture sharpness information is obtained based on third interaction information with the interaction page.

7. An image restoration device based on diffusion model and generative adversarial training, characterized in that: The method comprises: An image data acquisition module, configured to acquire image data and input the image data into a degradation removal encoder to obtain a first result; A restriction information acquisition module, configured to provide an interactive page and acquire image restriction information based on the interactive page; an image restoration processing module, configured to input the first result and the image limitation information into an image restoration network to obtain a second result, wherein the image restoration network is generated based on a diffusion model in a pre-trained diffusion network; The restoration result acquisition module is used to input the second result into the image decoder for processing, obtain the restoration result, and feed back.

8. A network device, characterized in that: include: A memory, a transceiver, and a processor; wherein the memory is used to store computer programs; the transceiver is used to send and receive data under the control of the processor; The processor is configured to read the computer program in the memory and execute the method according to claims 1-6.

9. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to claims 1-6 is implemented.

Images