Image denoising model generation method and device, electronic equipment and storage medium

Abstract

The invention provides an image denoising model generation method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used for denoising the image; pruning the initial network model to obtain a first network model, and distilling the initial network model to obtain a second network model; and generating an image denoising model according to the first network model and the second network model. According to the method provided by the invention, the speed of the image denoising process is improved, and the quality of the denoised image is improved.

Description

Technical Field

[0001] The present application relates to the field of image generation, and in particular to a method, device, electronic device and storage medium for generating an image denoising model. Background Art

[0002] The diffusion model is a generative model that uses an iterative denoising process to synthesize data. The inference process is the process of the diffusion model iteratively denoising the noisy image until a clear image is generated. The latent diffusion model is an improved version of the diffusion model. In recent years, the latent diffusion model has been widely used in image processing and generation, and can significantly improve the denoising effect and image generation efficiency.

[0003] However, the slow speed of the inference process has become a major obstacle to the generation of images by the latent diffusion model. Therefore, how to improve the speed of the latent diffusion model inference process has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage medium for generating an image denoising model, so as to solve the technical problem of slow speed of the latent diffusion model reasoning process.

[0005] In a first aspect, the present application provides a method for generating an image denoising model, comprising:

[0006] Obtaining an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image;

[0007] Performing pruning processing on the initial network model to obtain a first network model, and performing distillation processing on the initial network model to obtain a second network model;

[0008] An image denoising model is generated according to the first network model and the second network model.

[0009] Optionally, in the method described above, the initial network model includes multiple network layers; the initial network model is pruned to obtain a first network model, including:

[0010] Determine the input dimension and output dimension of the network layer in the initial network model; wherein the input dimension represents the number of data features of the input data, and the output dimension represents the feature vector data of the output data;

[0011] According to the input dimension and output dimension of each network layer in the initial network model, the initial network model is pruned to obtain a first network model.

[0012] Optionally, the method described above, pruning the initial network model according to the input dimension and output dimension of each network layer in the initial network model to obtain the first network model, includes:

[0013] According to the input dimension and output dimension of each network layer in the initial network model, determine the target layer from the network layers of the initial network model;

[0014] The target layer is pruned, and the initial network model after the pruning is determined as the first network model.

[0015] Optionally, the method as described above, determining the target layer from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model, includes:

[0016] According to the input dimension and output dimension of each network layer in the initial network model, determine the candidate layer from the network layer of the initial network model; wherein the input dimension of the candidate layer is the same and the output dimension of the candidate layer is the same;

[0017] According to a preset pruning ratio, the target layer is determined from the candidate layers; wherein the preset pruning ratio represents the proportion of the number of target layers to the number of networks.

[0018] Optionally, the method as described above further comprises:

[0019] Acquiring preset first image data;

[0020] Inputting the preset first image data into the network model to obtain output data corresponding to the initial network model, and inputting the preset first image data into the first network model to obtain output data corresponding to the first network model;

[0021] Determining a minimum average square error between output data corresponding to the initial network model and output data corresponding to the first network model;

[0022] If the minimum mean variance is no longer within the preset variance range, a new target layer is determined from the candidate layers according to a preset pruning ratio, and the new target layer is pruned to obtain a new first network model.

[0023] Optionally, in the method described above, the initial network model is distilled to obtain a second network model, including:

[0024] Based on the preset consistency distillation algorithm, the initial network model is distilled to obtain the second network model.

[0025] Optionally, the method described above, based on a preset consistency distillation algorithm, performs distillation processing on the initial network model to obtain a second network model, including:

[0026] The initial network model is determined as the teacher model, and the student model corresponding to the teacher model is determined based on the preset consistency distillation algorithm;

[0027] Obtaining preset second image data, inputting the preset second image data into the teacher model, and obtaining output data corresponding to the teacher model;

[0028] The student model is trained according to the preset second image data and the output data corresponding to the teacher model to obtain a trained second network model.

[0029] Optionally, the method as described above, generating an image denoising model according to the first network model and the second network model, comprises:

[0030] Obtain a preset first weight and a preset second weight; wherein the preset first weight represents the importance of pruning, and the preset second weight represents the importance of distillation;

[0031] According to the preset first weight and the preset second weight, the first network model and the second network model are fused to obtain an image denoising model.

[0032] Optionally, the method as described above further comprises:

[0033] Obtain image data to be denoised;

[0034] The image data to be denoised is input into the image denoising model to obtain the output denoised image data.

[0035] In a second aspect, the present application provides a device for generating an image denoising model, comprising:

[0036] An acquisition unit, used for acquiring an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used for performing denoising processing on the image;

[0037] A processing unit, configured to perform pruning processing on the initial network model to obtain a first network model, and perform distillation processing on the initial network model to obtain a second network model;

[0038] A generating unit is used to generate an image denoising model according to the first network model and the second network model.

[0039] In a third aspect, an electronic device is provided, including:

[0040] at least one processor, and a memory in communication with the processor;

[0041] The memory stores computer-executable instructions;

[0042] The processor executes the computer-executable instructions stored in the memory to implement the method described in any one of the first aspects.

[0043] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any one of the methods described in the first aspect.

[0044] According to a fifth aspect, a computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method described in any one of the first aspects is implemented.

[0045] The method, device, electronic device and storage medium for generating an image denoising model provided by the present application obtain a first network model by pruning a preset initial network model, obtain a second network model by distilling the preset initial network model, and generate an image denoising model based on the first network model and the second network model. Through pruning and distillation, the complexity of the initial network model is reduced and the denoising efficiency of the initial network model is improved; through the fusion of the first network model and the second network model, the effect and quality of the image denoising model are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] Figure 1 A schematic diagram of a flow chart of a method for generating an image denoising model provided in an embodiment of the present disclosure;

[0048] Figure 2 A schematic diagram of a flow chart of a method for generating an image denoising model provided in an embodiment of the present disclosure;

[0049] Figure 3 A schematic diagram of a flow chart of a method for generating an image denoising model provided in an embodiment of the present disclosure;

[0050] Figure 4 A schematic diagram of a flow chart of a method for generating an image denoising model provided in an embodiment of the present disclosure;

[0051] Figure 5 A structural block diagram of a device for generating an image denoising model provided by an embodiment of the present disclosure;

[0052] Figure 6 A structural block diagram of a device for generating an image denoising model provided by an embodiment of the present disclosure;

[0053] Figure 7A structural block diagram of an electronic device provided for an embodiment of the present disclosure;

[0054] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment.

[0055] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] First, the terms involved in this application are explained:

[0059] Diffusion model: Diffusion model is a generative model whose core idea is to corrupt the training data by continuously adding Gaussian noise, and then restore the data by learning to invert this noise process.

[0060] Latent Diffusion Models: Latent Diffusion Models (LDM) are an improved version of the diffusion model that applies the diffusion process to the latent space of the image instead of the pixel space. This approach improves the generation efficiency and allows the generation of higher resolution images.

[0061] Stable-Diffusion-XL-base-1.0 (SDXL) model: Stable-Diffusion (SD for short) is a multimodal open source generative model based on Latent Diffusion Models (LDMs), which can synthesize high-resolution images based on given text prompts. Stable-Diffusion-XL is a two-stage cascade diffusion model (Latent Diffusion Model) based on SD, including Base model and Refiner model. The main work of the Base model is consistent with Stable Diffusion 1.x-2.x, and has the capabilities of text-to-image and image-to-image.

[0062] Latent space: Latent space is a low-dimensional space used to represent data in machine learning. It refers to a compressed representation of all the useful information contained in the data. The latent space is usually lower dimensional than the original data space, which makes it easier to analyze and understand.

[0063] Model pruning: The process of removing connections or neuron nodes in the model whose weights are less than a certain threshold to obtain a sparser model network. Model pruning is mainly divided into structured pruning and unstructured pruning.

[0064] Structured pruning: Structured pruning usually uses filters or entire network layers as basic units for pruning. When a filter is pruned, its previous feature map and the next feature map will change accordingly, but the structure of the model is not destroyed and can still be accelerated by GPU or other hardware. Therefore, this type of method is called structured pruning.

[0065] Unstructured pruning: Unstructured pruning removes unimportant neurons. Accordingly, the connections between the pruned neurons and other neurons are ignored during calculation. Since the pruned model is usually sparse and destroys the structure of the original model, this type of method is called unstructured pruning.

[0066] UNet: UNet is a special convolutional neural network that was originally designed for biomedical image segmentation. Its structure makes UNet the preferred architecture for many image generation tasks, including diffusion models. The UNet model adds noise to the original image, uses noise as a label, and continuously learns to reduce the loss value of predicted noise and actual noise.

[0067] Consistency Model: The consistency model (CM) adds a constraint based on the diffusion model.

[0068] Latent Consistency Model: The Latent Consistency Model (LCM) generates high-resolution images by adopting a consistency model in the image latent space, reducing the need for a large number of iterations and achieving fast and high-fidelity sampling.

[0069] Latent Consistency Distillation: The process of distilling the latent diffusion model (LDM) into a latent consistency model (LCM) is called latent consistency distillation (LCD).

[0070] LAION-5B: LAION-5B is a large-scale multimodal dataset containing 5.85 billion image-text pairs, including multiple subsets such as English image-text pairs, image-text pairs in multiple languages, and samples in undetectable languages.

[0071] LCM-LoRA: By applying LoRA to latent consistency distillation (LCD), the LCM-LoRA module is obtained. It is a general training-free acceleration module that can be directly inserted into various Stable-Diffusion (SD) fine-tuning models or SD LoRA to support fast reasoning with minimal steps.

[0072] LoRA: The full name of the LoRA model is: Low-Rank Adaptation of Large Language Models. It can be understood as a plug-in, a model that only requires a small amount of data to be trained. When generating images, the LoRA model will be used in conjunction with the large model to adjust the output image results.

[0073] CLIP score: CLIP score measures the text-image relevance, the higher the score, the higher the relevance. Text-image relevance is considered to be the semantic similarity between text and image.

[0074] FID: FID is used to measure the similarity between two image datasets. FID first calculates the Inception feature vectors of the real image and the generated image, then calculates the mean, covariance and other statistics of the feature vectors, and then calculates the distance based on these statistics. The smaller the FID, the better the quality of the generated image.

[0075] The diffusion model is a generative model that uses an iterative denoising process to synthesize data. The inference process is the process of the diffusion model iteratively denoising the noisy image until a clear image is generated. The latent diffusion model is an improved version of the diffusion model. The iterative denoising step accounts for a large proportion of the time consumed in a complete inference process. The slowness of the iterative denoising step limits the practicality of the diffusion model.

[0076] At present, the latent diffusion model is widely used in the field of image generation as an improved diffusion model. How to accelerate the reasoning process of the latent diffusion model has become an urgent problem to be solved. The main acceleration methods currently include model pruning and model distillation, both of which compress the original latent diffusion model. However, each single method has its own defects. For example, in order to achieve better performance, the previous model pruning requires manual identification of unimportant layers and manual removal, and the degree of automation is low; the traditional model distillation method involves the training of teacher models and student models, which requires additional computing resources. Therefore, how to balance the model compression amount and the model prediction accuracy is an important difficulty.

[0077] The specific application scenario of the present application is the generation of a latent diffusion model, which is used to prune the trained latent model to obtain a pruned latent diffusion model, and to distill the trained latent diffusion model to obtain a student model of the latent diffusion model, and to fuse the pruned latent diffusion model with the student model to generate a new model.

[0078] The data transmission method provided in this application is intended to solve the above technical problems in the prior art.

[0079] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0080] Figure 1 A schematic flow chart of a method for generating an image denoising model provided in an embodiment of the present disclosure, the method can be executed by a device for generating an image denoising model.

[0081] like Figure 1 As shown, the method comprises the following steps:

[0082] S101, obtaining an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image.

[0083] Exemplarily, an initial network model is obtained. Among them, the initial network model is a preset diffusion model, and the diffusion model is a generative model. By continuously adding random noise to the data, and then learning to reverse the diffusion process, data is constructed from the noise. The diffusion model can be used to denoise the noisy image and generate the image. The initial network model is a diffusion model that is first constructed and trained. It can be a potential diffusion model, for example, a stable-diffusion-xl-base-1.0 (SDXL) model. The potential diffusion model is an improved diffusion model. The potential diffusion model performs an iterative denoising process in the latent space, and converts the image from the latent space to the pixel space through a decoder. The decoder is used to decode the image from the latent space. The decoder can be, for example, a VAE (variational autoencoder). In this embodiment, the model structure of the initial network model is not specifically limited.

[0084] S102: Pruning the initial network model to obtain a first network model, and distilling the initial network model to obtain a second network model.

[0085] Exemplarily, model pruning rules are pre-set, and the model pruning rules are used to prune the initial network model. Pruning is to remove redundant or unimportant channels, parameters, etc. in the initial network model to improve the efficiency and speed of the initial network model. The pruning rules can be, for example, one-time pruning rules, that is, the importance of the structure in the initial network model is evaluated, and after the evaluation is completed, the parts with low importance are pruned. The structure in the initial network model can be a network layer in the model, and the pruning process can be, for example, a structured pruning method, that is, pruning is performed with filters or the entire network layer in the network model as the basic unit.

[0086] The model distillation rule is pre-set, and the model distillation rule is used to obtain a model that is close to the initial network model, has small parameters, weak learning ability, and fast reasoning speed according to the initial network model. The model distillation rule can be, for example, taking the initial network model as the teacher model, and determining a simplified potential diffusion model as the student model, and training the teacher model on a pre-prepared data set to obtain at least one soft label, and passing the soft label to the student model, and using the soft label and the pre-prepared data set to train the student model, so that the prediction result of the student model is close to the prediction result of the teacher model. Among them, the data set can include multiple images, and the soft label refers to the label output by the teacher model. The soft label contains the probability distribution information about each data feature, so that each data feature has a probability value distribution between 0 and 1, and the probability value distribution is used to characterize the accuracy of the prediction. The data feature can be, for example, color, shape, texture, etc. The probability value distribution can be, for example, for a certain type of noise, the image generated after denoising the noise has a probability of 0.3 for the color being red and a probability of 0.7 for the color being blue.

[0087] According to the preset model pruning rules, the initial network model is pruned to obtain the first network model. A data set is prepared in advance, and the data set is used to train the teacher model and the student model. The data set can be a LAION-5B data set. According to the preset model distillation rules, the initial network model is determined to be the teacher model, and the student model corresponding to the teacher model is determined. The teacher model is trained using the pre-prepared data set to obtain the soft labels learned by the teacher model, and the soft labels are passed to the student model. The student model is trained using the soft labels passed to the student model by the teacher model and the pre-prepared data set, so that the prediction results of the student model are close to the prediction results of the teacher model. The trained student model is the second network model.

[0088] S103: Generate an image denoising model according to the first network model and the second network model.

[0089] Exemplarily, a preset model loading method is used to load the first network model and the second network model to obtain an image denoising model. The preset model loading method is used to load the first network model and the second network model. The preset model loading method may be to pre-set the weights of the first network model and the second network model, and fuse the first network model and the second network model according to the weights. In this embodiment, the model loading method is not specifically limited.

[0090] The method, device, electronic device and storage medium for generating an image denoising model provided by the present application obtain a first network model by pruning a preset initial network model, obtain a second network model by distilling the preset initial network model, and generate an image denoising model based on the first network model and the second network model. Through pruning and distillation, the complexity of the initial network model is reduced and the denoising efficiency of the initial network model is improved; through the fusion of the first network model and the second network model, the effect and quality of the image denoising model are improved.

[0091] Figure 2 A flowchart of a method for generating an image denoising model provided in an embodiment of the present disclosure.

[0092] In this embodiment, the initial network model includes multiple network layers; the initial network model is pruned to obtain a first network model, including: determining the input dimension and output dimension of the network layer in the initial network model; wherein the input dimension represents the number of data features of the input data, and the output dimension represents the feature vector data of the output data; according to the input dimension and output dimension of each network layer in the initial network model, the network initial model is pruned to obtain the first network model.

[0093] like Figure 2 As shown, the method comprises the following steps:

[0094] S201, obtaining an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image.

[0095] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0096] S202, determining the input dimension and output dimension of the network layer in the initial network model; wherein the input dimension represents the number of data features of the input data, and the output dimension represents the feature vector data of the output data.

[0097] Exemplarily, the UNet model is an important component of the initial network model, which is mainly used to predict noise. The UNet model can be regarded as a combination of multiple network layers, and the network layer represents the layer structure in the initial network model. The network layer can be, for example, a residual layer, a Transformer layer, and a convolutional layer. For each network layer, there are input dimensions and output dimensions corresponding to each network layer. The input dimension represents the number of data features of the input data; the output dimension represents the feature vector data of the output data. Determine the input dimension and output dimension of each network layer in the initial network model.

[0098] The beneficial effect of this setting is that the input dimension and output dimension of each network layer of the UNet model in the initial network model are determined, making the input-output relationship of each network layer more intuitive and facilitating the subsequent processing of each network layer according to the input dimension and output dimension.

[0099] S203. According to the input dimension and output dimension of each network layer in the initial network model, the initial network model is pruned to obtain a first network model, and the initial network model is distilled to obtain a second network model.

[0100] Exemplarily, the preset model pruning rules in this embodiment can be, for example, presetting the input dimension and output dimension of the network layer to be pruned, and performing structured pruning on the initial network model, that is, pruning the network layer with the same input dimension as the preset and the same output dimension as the preset. After obtaining the input dimension and output dimension of each network layer, the initial network model is pruned according to the preset model pruning rules to obtain the first network model. The initial network model is distilled according to the preset model distillation rules to obtain the second network model.

[0101] The beneficial effect of this setting is that, according to the input dimension and output dimension of each network layer, the network layer whose input dimension and output dimension meet the preset requirements is pruned, which better reduces the performance degradation of the network model caused by pruning and ensures the performance of the network model after pruning.

[0102] In this embodiment, the network initial model is pruned according to the input dimensions and output dimensions of each network layer in the initial network model to obtain a first network model, including: determining a target layer from the network layer of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model; pruning the target layer, and determining the pruned initial network model as the first network model.

[0103] Specifically, a target layer determination rule is set in advance. The target layer determination rule is used to determine the target layer from each network layer according to the input dimension and output dimension of each network layer. The target layer determination rule may be to pre-set the input dimension standard and the output dimension standard of the target layer. The input dimension standard and the output dimension standard represent the input dimension and output dimension that the required target layer should have. The network layer whose input dimension is the same as the input dimension standard and whose output dimension is the same as the output dimension standard is determined as the target layer.

[0104] According to the input dimension and output dimension of each network layer in the initial network model, based on the preset target layer determination rule, the target layer is determined from the network layer of the initial network model, and the target layer is pruned, and the pruned initial network model is determined as the first network model.

[0105] The beneficial effect of this setting is that the input dimension standard and output dimension standard of the target layer are set according to the needs, and the network layer that meets the standard is determined as the target layer, which makes the pruning process more accurate and more automated, and improves the effect of model pruning.

[0106] In this embodiment, a target layer is determined from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model, including: determining a candidate layer from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model; wherein the input dimensions of the candidate layers are the same and the output dimensions of the candidate layers are the same; determining the target layer from the candidate layers according to a preset pruning ratio; wherein the preset pruning ratio represents the proportion of the number of target layers to the number of network layers.

[0107] Specifically, after determining the input dimension and output dimension of each network layer, for the same network layer, if the input dimension and output dimension of the network layer are the same, the network layer is determined as a candidate layer. The candidate layer is determined from the network layers of the initial network model. Among them, the candidate layer is a network layer with the same input dimension and output dimension, and the candidate layer is used to determine the target layer from the candidate layers.

[0108] The pruning ratio is pre-set, and the pruning ratio represents the ratio of the number of target layers to the number of network layers, that is, the ratio of the layers to be pruned to the value of the network layers. If the pruning ratio should be less than the ratio of candidate layers to network layers, then according to the preset pruning ratio, the candidate layers that meet the ratio are randomly selected from the candidate layers as the target layers. If the pruning ratio is greater than the ratio of candidate layers to network layers, all candidate layers are pruned.

[0109] The beneficial effect of this setting is that multiple candidate layers are determined according to the input and output dimensions, and the target layer is determined from the candidate layers according to the preset pruning ratio, which makes the pruning process more accurate and improves the diversity of the pruning process, facilitates the subsequent processing and adjustment of the pruned model with poor results, and improves the flexibility of the pruning process.

[0110] In this embodiment, it also includes: obtaining preset first image data; inputting the preset first image data into the initial network model to obtain output data corresponding to the initial network model, and inputting the preset first image data into the first network model to obtain output data corresponding to the first network model; determining the minimum mean variance between the output data corresponding to the initial network model and the output data corresponding to the first network model; if the minimum mean variance is not within the preset variance range, determining a new target layer from the candidate layer according to a preset pruning ratio, and pruning the new target layer to obtain a new first network model.

[0111] Specifically, after the initial network model is pruned to obtain the first network model. Obtain preset first image data, the first image data is used to test the performance of the first network model, and the first image data can be at least one noisy picture. Input the preset first image data into the initial network model, use the initial network model to infer the first image data, and iteratively denoise the first image data to obtain generated image data, which is the output data corresponding to the initial network model; input the preset first image data into the first network model, use the first network model to infer the first image data, and iteratively denoise the first image data to obtain generated image data, which is the output data corresponding to the first network model.

[0112] A variance determination rule is pre-set, and the variance determination rule is used to determine the minimum average variance between the output data corresponding to the initial network model and the output data corresponding to the first network model. The variance determination rule can be a dynamic programming method. A variance range is pre-set, and the upper limit of the variance range can be the variance corresponding to the performance loss threshold preset by the first network model for the initial network model, and the lower limit of the variance range can be 0. Among them, the performance loss is the value of the performance degradation of the first network model compared with the initial network model when denoising the noisy image to generate an image. In order to prevent the performance of the first network model from being too degraded, a performance loss threshold is set for the performance loss. When the performance loss value of the first network model exceeds the performance loss threshold, the first network model does not meet the requirements. According to the output data corresponding to the initial network model and the output data corresponding to the first network model, based on the preset variance determination rule, the minimum average variance between the output data corresponding to the initial network model and the output data corresponding to the first network model is determined. If the minimum average variance is not within the preset variance range, a new target layer is determined from the candidate layer according to the pruning ratio, and the new target layer is pruned to obtain a new first network model.

[0113] The beneficial effect of such a setting is that the performance of the initial network model and the performance of the first network model are verified using the same image data, and the minimum average variance that characterizes the performance loss of the first network model compared to the initial network model is obtained, and based on the minimum average variance, it is determined whether the initial network model needs to be pruned again. This makes the reliability of the pruned first network model higher, and the unqualified first network model can be re-pruned and adjusted in time, so that the lower limit of the first network model is improved, and the performance of the first network model is guaranteed.

[0114] S204: Generate an image denoising model according to the first network model and the second network model.

[0115] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0116] The method, device, electronic device and storage medium for generating an image denoising model provided by the present application obtain a first network model by pruning a preset initial network model, obtain a second network model by distilling the preset initial network model, and generate an image denoising model based on the first network model and the second network model. Through pruning and distillation, the complexity of the initial network model is reduced and the denoising efficiency of the initial network model is improved; through the fusion of the first network model and the second network model, the effect and quality of the image denoising model are improved.

[0117] Figure 3 A flowchart of a method for generating an image denoising model provided in an embodiment of the present disclosure.

[0118] In this embodiment, the initial network model is distilled to obtain the second network model, including: based on a preset consistency distillation algorithm, the initial network model is distilled to obtain the second network model.

[0119] like Figure 3 As shown, the method comprises the following steps:

[0120] S301, obtaining an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image.

[0121] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0122] S302: Prune the initial network model to obtain a first network model, and distill the initial network model based on a preset consistency distillation algorithm to obtain a second network model.

[0123] Exemplarily, according to the preset model pruning rules, the initial network model is pruned to obtain the first network model. A consistency distillation algorithm is pre-set, and the consistency distillation algorithm is used to distill the initial network model. The preset consistency distillation algorithm is used to distill the potential diffusion model into a potential consistency model. The preset consistency distillation algorithm can be a potential consistency distillation method, i.e., LCD. Based on the preset consistency distillation algorithm, the initial network model is distilled to obtain the second network model.

[0124] The beneficial effect of this setting is that the latent diffusion model is distilled into a latent consistency model, which reduces the complexity of the model, speeds up the model's generation of images, and significantly improves the speed of the iterative denoising step, thereby speeding up the inference process and improving the performance of the model.

[0125] In this embodiment, the initial network model is distilled based on a preset consistency distillation algorithm to obtain a second network model, including: determining the initial network model as a teacher model, and determining a student model corresponding to the teacher model based on a preset consistency distillation algorithm; obtaining preset second image data, inputting the preset second image data into the teacher model, and obtaining output data corresponding to the teacher model; training the student model according to the preset second image data and the output data corresponding to the teacher model to obtain a trained second network model.

[0126] Specifically, the initial network model is determined as the teacher model. Based on the preset consistency distillation algorithm, the student model corresponding to the teacher model is determined, and the student model is used to learn and accelerate the reasoning process of the teacher model. The preset second image data is obtained, and the second image data is used to train the teacher model and the student model during the distillation process. The second image data can be, for example, a subset of data with an aesthetic score of more than 6 points in the LAION-5B dataset. The preset second image data is input into the teacher model, the output data corresponding to the teacher model is obtained, at least one soft label is obtained, the soft label is passed to the student model, and the student model is trained according to the preset second image data and the soft label passed by the teacher model.

[0127] During the training process, LoRA technology is introduced to reduce the number of training parameters during student model training, thereby accelerating the training process. Through LoRA technology, at least one trainable network layer, namely the LoRA layer, is inserted into the student model. According to the preset second image data and the soft label transmitted by the teacher model, the trainable network layer in the student model is trained so that the output data of the student model approaches the output data of the teacher model, and the trained student model, namely the LCM-LoRA model, is obtained as the second network model.

[0128] The beneficial effect of this setting is that by using potential consistency distillation, the distillation process of the initial network model is faster and more efficient, which can speed up the inference speed of the second network model. By training with a subset of data with an aesthetic score of more than 6 points in the LAION-5B dataset, the performance gap between the second network model and the first network model can be made up. By using LoRA technology, the distillation process is accelerated and resource consumption is reduced, and the generated LCM-LoRA model is also more versatile.

[0129] S303: Generate an image denoising model according to the first network model and the second network model.

[0130] Exemplarily, this step can refer to the above-mentioned step S103 and will not be described in detail.

[0131] The method, device, electronic device and storage medium for generating an image denoising model provided by the present application obtain a first network model by pruning a preset initial network model, obtain a second network model by distilling the preset initial network model, and generate an image denoising model based on the first network model and the second network model. Through pruning and distillation, the complexity of the initial network model is reduced and the denoising efficiency of the initial network model is improved; through the fusion of the first network model and the second network model, the effect and quality of the image denoising model are improved.

[0132] Figure 4 A flowchart of a method for generating an image denoising model provided in an embodiment of the present disclosure.

[0133] In this embodiment, an image denoising model is generated according to the first network model and the second network model, including: obtaining a preset first weight and a preset second weight; wherein the preset first weight represents the importance of pruning, and the preset second weight represents the importance of distillation; according to the preset first weight and the preset second weight, the first network model and the second network model are fused to obtain the image denoising model.

[0134] like Figure 4 As shown, the method comprises the following steps:

[0135] S401, obtaining an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image.

[0136] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0137] S402: Pruning the initial network model to obtain a first network model, and distilling the initial network model to obtain a second network model.

[0138] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0139] S403, obtaining a preset first weight and a preset second weight; wherein the preset first weight represents the importance of pruning, and the preset second weight represents the importance of distillation.

[0140] Exemplarily, a preset first weight and a preset second weight are obtained, wherein the preset first weight represents the importance of the first network model obtained through model pruning, and the preset second weight represents the importance of the second network model obtained through feature model distillation. The preset first weight and the preset second weight are used to load the first network model and the second network model in combination.

[0141] The beneficial effect of such a setting is that, according to the needs of different usage scenarios, different weights can be set for the first network model and the second network model, which facilitates loading the network models according to different weights, making the final model more in line with the needs of the actual scenario, and facilitates using weights to adjust and optimize the model according to the final effect.

[0142] S404: According to the preset first weight and the preset second weight, the first network model and the second network model are fused to obtain an image denoising model.

[0143] Exemplarily, a preset loading method is used to fuse the first network model and the second network model according to the preset first weight and the preset second weight, that is, the first network model and the second network model are superimposed and loaded according to the preset first weight and the preset second weight to generate an image denoising model. The image denoising model is the final model in which the first network model and the second network model are fused according to the first weight and the second weight. The preset loading method can be the loading method provided by the diffusers module.

[0144] The beneficial effect of this setting is that it can fully combine the advantages of the first network model and the second network model, simplify the model while improving the speed of iterative denoising, thereby improving the model's reasoning speed and reasoning effect, and effectively reducing memory usage. The second network model that has been distilled can make up for the performance loss of the first network model caused by pruning.

[0145] In this embodiment, the method further includes: obtaining image data to be denoised; inputting the image data to be denoised into an image denoising model to obtain output denoised image data.

[0146] Specifically, the image data to be denoised is obtained, and the image data to be denoised is used to input the model. The image data to be denoised can be a pure noise image. After the image data to be denoised is input, a text description of the denoised image data is input to the model, and the model is made to infer the denoised data and generate an image based on the text description. The text description represents the text description of the image required by the user.

[0147] Input the image data to be denoised into the initial network model, input a text description of the denoised image data into the initial network model, and obtain the denoised image data corresponding to the initial network model. Input the image data to be denoised into the first network model, input a text description of the denoised image data into the first network model, and obtain the denoised image data corresponding to the first network model. Input the image data to be denoised into the second network model, input a text description of the denoised image data into the second network model, and obtain the denoised image data corresponding to the second network model. Input the image data to be denoised into the image denoising model, input a text description of the denoised image data into the image denoising model, and obtain the denoised image data corresponding to the image denoising model.

[0148] The scoring indicators are pre-set and used to score the performance of the model. The scoring indicators can be FID and CLIP indicators. The CLIP indicator represents the text-image association. The higher the CLIP score, the higher the association and the better the model performance. The FID represents the similarity between the two data sets. The smaller the FID, the better the quality of the generated image and the better the model performance. Use the preset scoring indicators to score the denoised image data corresponding to the initial network model to obtain the CLIP score and FID of the initial network model.

[0149] Since the model pruning process will cause the performance of the first network model to be lower than that of the initial network model, in order to limit the performance loss of the first network model compared to the initial network model, the CLIP score threshold and FID threshold of the first network model are preset, wherein the CLIP score threshold represents the lower limit of the allowable CLIP score threshold of the first network model; the FID threshold represents the upper limit of the allowable FID threshold of the first network model. If the CLIP score of the first network model is less than the preset CLIP score threshold, or the FID of the first network model is greater than the preset FID threshold, the initial network model is pruned again, the target layer is reselected from the candidate layer, and pruned to obtain a new first network model.

[0150] The CLIP score threshold and FID threshold of the second network model are preset, wherein the CLIP score threshold represents the lower limit of the permissible CLIP score threshold of the second network model; the FID threshold represents the upper limit of the permissible FID threshold of the second network model. If the CLIP score of the second network model is less than the preset CLIP score threshold, or the FID of the second network model is greater than the preset FID threshold, the initial network model is re-distilled, and the student model is re-trained using the preset second image data to obtain a new second network model.

[0151] The CLIP score threshold and FID threshold of the image denoising model are preset, wherein the CLIP score threshold represents the lower limit of the CLIP score threshold of the image denoising model that can be allowed; the FID threshold represents the upper limit of the FID threshold of the image denoising model that can be allowed. If the CLIP score of the image denoising model is less than the preset CLIP score threshold, or the FID of the image denoising model is greater than the preset FID threshold, the preset first weight and the second weight are adjusted. For example, if the current first weight is greater than the second weight, the first weight is reduced and the second weight is increased. If the number of adjustments to the preset first weight and the second weight exceeds the preset number, the initial network model is re-pruned to obtain a new first network model; or the initial network model is re-distilled to obtain a new second network model.

[0152] The beneficial effect of this setting is that the performance of each model is scored according to the image data to be denoised, and the models whose scores do not meet the requirements can be adjusted in real time, making the processing of the model more flexible, making the performance of the final model stronger, and better accelerating the inference process.

[0153] The method, device, electronic device and storage medium for generating an image denoising model provided by the present application obtain a first network model by pruning a preset initial network model, obtain a second network model by distilling the preset initial network model, and generate an image denoising model based on the first network model and the second network model. Through pruning and distillation, the complexity of the initial network model is reduced and the denoising efficiency of the initial network model is improved; through the fusion of the first network model and the second network model, the effect and quality of the image denoising model are improved.

[0154] Figure 5 A structural block diagram of a device for generating an image denoising model provided in an embodiment of the present disclosure.

[0155] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Figure 5 The image denoising model generation device 500 includes: an acquisition unit 501, a processing unit 502 and a generation unit 503.

[0156] An acquisition unit 501 is used to acquire an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image;

[0157] The processing unit 502 is used to perform pruning processing on the initial network model to obtain a first network model, and to perform distillation processing on the initial network model to obtain a second network model;

[0158] The generating unit 503 is used to generate an image denoising model according to the first network model and the second network model.

[0159] Figure 6 A structural block diagram of a device for generating an image denoising model provided in an embodiment of the present disclosure.

[0160] exist Figure 5 Based on the embodiment shown, the initial network model includes multiple network layers, such as Figure 6 As shown, the processing unit 502 includes a determination module 5021 and a pruning module 5022 .

[0161] A determination module 5021 is used to determine the input dimension and output dimension of the network layer in the initial network model; wherein the input dimension represents the number of data features of the input data, and the output dimension represents the feature vector data of the output data;

[0162] The pruning module 5022 is used to prune the initial network model according to the input dimension and output dimension of each network layer in the initial network model to obtain the first network model.

[0163] In one example, the pruning module 5022 includes:

[0164] A target layer determination submodule, used to determine a target layer from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model;

[0165] The pruning submodule is used to perform pruning on the target layer and determine the initial network model after the pruning as the first network model.

[0166] In one example, the target layer determines the submodule, which is specifically used for:

[0167] Determine candidate layers from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model; wherein the input dimensions of the candidate layers are the same and the output dimensions of the candidate layers are the same;

[0168] According to a preset pruning ratio, the target layer is determined from the candidate layers; wherein the preset pruning ratio represents the proportion of the number of target layers to the number of network layers.

[0169] In one example, the device further includes:

[0170] A first acquisition unit, used to acquire preset first image data;

[0171] An output unit, used to input the preset first image data into the network model to obtain output data corresponding to the initial network model, and to input the preset first image data into the first network model to obtain output data corresponding to the first network model;

[0172] a variance determination unit, configured to determine a minimum average variance between output data corresponding to the initial network model and output data corresponding to the first network model;

[0173] A pruning unit is used to determine a new target layer from the candidate layers according to a preset pruning ratio if the minimum mean variance is no longer within a preset variance range, and to prune the new target layer to obtain a new first network model.

[0174] In one example, the processing unit 502 further includes:

[0175] The distillation module 5023 is used to perform distillation processing on the initial network model based on a preset consistency distillation algorithm to obtain the second network model.

[0176] In one example, the distillation module 5023 includes:

[0177] A model determination submodule, used to determine the initial network model as a teacher model, and determine a student model corresponding to the teacher model based on a preset consistency distillation algorithm;

[0178] A first training submodule is used to obtain preset second image data, input the preset second image data into the teacher model, and obtain output data corresponding to the teacher model;

[0179] The second training submodule is used to train the student model according to the preset second image data and the output data corresponding to the teacher model to obtain a trained second network model.

[0180] In one example, the generating unit 503 includes:

[0181] A weight module, used to obtain a preset first weight and a preset second weight; wherein the preset first weight represents the importance of pruning, and the preset second weight represents the importance of distillation;

[0182] A fusion module is used to fuse the first network model and the second network model according to the preset first weight and the preset second weight to obtain the image denoising model.

[0183] In one example, it also includes:

[0184] A second acquisition unit, used to acquire image data to be denoised;

[0185] The image output unit is used to input the image data to be denoised into the image denoising model to obtain output denoised image data.

[0186] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, the electronic device may be a terminal device or a server, such as Figure 7 As shown, the electronic device 700 includes: at least one processor 702; and a memory 701 communicatively connected to the at least one processor 702; wherein the memory stores instructions executable by the at least one processor 702, and the instructions are executed by the at least one processor 702 so that the at least one processor 702 can execute the method for generating an image denoising model disclosed in the present invention.

[0187] The electronic device 700 further includes a receiver 703 and a transmitter 704. The receiver 703 is used to receive instructions and data sent by other devices, and the transmitter 704 is used to send instructions and data to external devices.

[0188] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0189] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0190] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment, which device may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0191] Device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 811 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0192] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0193] The memory 804 is configured to store various types of data to support operations on the device 700. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0194] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

[0195] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0196] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0197] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0198] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also detect the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0199] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0200] In an exemplary embodiment, the device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0201] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0202] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal device, enables the terminal device to execute the above-mentioned method for generating an image denoising model.

[0203] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0204] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0205] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0206] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0207] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0208] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0209] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0210] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0211] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating an image denoising model, characterized in that: The method includes: Acquire an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used to perform denoising on the image; Performing pruning processing on the initial network model to obtain a first network model, and performing distillation processing on the initial network model to obtain a second network model; An image denoising model is generated according to the first network model and the second network model.

2. The method according to claim 1, characterized in that The initial network model includes a plurality of network layers; the initial network model is pruned to obtain a first network model, including: Determine the input dimension and output dimension of the network layer in the initial network model; wherein the input dimension represents the number of data features of the input data, and the output dimension represents the feature vector data of the output data; The initial network model is pruned according to the input dimension and the output dimension of each network layer in the initial network model to obtain the first network model.

3. The method according to claim 2, characterized in that According to the input dimension and the output dimension of each network layer in the initial network model, the initial network model is pruned to obtain the first network model, including: Determining a target layer from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model; The target layer is pruned, and the initial network model after the pruning is determined as the first network model.

4. The method according to claim 3, characterized in that Determining a target layer from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model includes: Determine candidate layers from the network layers of the initial network model according to the input dimensions and output dimensions of each network layer in the initial network model; wherein the input dimensions of the candidate layers are the same and the output dimensions of the candidate layers are the same; According to a preset pruning ratio, the target layer is determined from the candidate layers; wherein the preset pruning ratio represents the proportion of the number of target layers to the number of network layers.

5. The method according to claim 4, characterized in that Also includes: Acquiring preset first image data; Inputting the preset first image data into the initial network model to obtain output data corresponding to the initial network model, and inputting the preset first image data into the first network model to obtain output data corresponding to the first network model; Determining a minimum average square error between output data corresponding to the initial network model and output data corresponding to the first network model; If the minimum mean variance is not within a preset variance range, a new target layer is determined from the candidate layers according to a preset pruning ratio, and the new target layer is pruned to obtain a new first network model.

6. The method according to claim 1, characterized in that The initial network model is distilled to obtain a second network model, including: Based on a preset consistency distillation algorithm, the initial network model is distilled to obtain the second network model.

7. The method according to claim 6, characterized in that Based on a preset consistency distillation algorithm, the initial network model is distilled to obtain the second network model, including: Determine the initial network model as a teacher model, and determine a student model corresponding to the teacher model based on a preset consistency distillation algorithm; Acquire preset second image data, input the preset second image data into the teacher model, and obtain output data corresponding to the teacher model; The student model is trained according to the preset second image data and the output data corresponding to the teacher model to obtain a trained second network model.

8. The method according to claim 1, characterized in that Generating an image denoising model according to the first network model and the second network model, including: Obtaining a preset first weight and a preset second weight; wherein the preset first weight represents the importance of pruning, and the preset second weight represents the importance of distillation; According to the preset first weight and the preset second weight, the first network model and the second network model are fused to obtain the image denoising model.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: Obtain image data to be denoised; The image data to be denoised is input into the image denoising model to obtain output denoised image data.

10. A device for generating an image denoising model, characterized in that: include: An acquisition unit, used for acquiring an initial network model; wherein the initial network model is a preset diffusion model, and the initial network model is used for performing denoising processing on the image; A processing unit, configured to perform pruning processing on the initial network model to obtain a first network model, and perform distillation processing on the initial network model to obtain a second network model; A generating unit is used to generate an image denoising model according to the first network model and the second network model.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images