Sketch progressive image translation method and system based on conditional diffusion model

Abstract

The application discloses a sketch progressive image translation method and system based on a conditional diffusion model, and the method comprises the following steps: inputting a to-be-processed sketch into a grayscale image generation model to obtain a sketch grayscale image, wherein the grayscale image generation model comprises a latent space unit and a conditional diffusion noise unit; performing feature extraction on the to-be-processed sketch and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector; performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information; and performing image translation on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model comprises a forward diffusion unit and a reverse diffusion unit. The sketch feature is combined with the generated grayscale image, and then is added to the translation process from the grayscale image to the real image, so that the accurate understanding of the overall structure and the local feature of the sketch can be excellently maintained, and the quality of the image translation is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a sketch progressive image translation method and system based on a conditional diffusion model. Background Art

[0002] With the continuous advancement of computer vision and natural language processing, image translation has attracted widespread attention as a key research task. Image translation aims to convert images between different modalities, such as converting an RGB image to a thermal image or translating a sketch into its corresponding real-world image. This task is crucial for achieving cross-modal information conversion and can provide solutions for many practical applications, such as virtual reality, assisted medical diagnosis, and artistic creation.

[0003] In recent years, methods based on generative adversarial networks (GANs) and diffusion have made significant progress in image translation. However, GAN-based models such as CycleGAN and SketchyGAN suffer from vulnerabilities such as vanishing gradients, mode collapse, and poor generalization. Diffusion-based models such as UNIT-DDPM are prone to pixel miscoloring. Image translation tasks, particularly those involving translating sketches into real images, still face numerous challenges.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a sketch progressive image translation method and system based on the conditional diffusion model, aiming to solve the technical problem of how to accurately translate sketches into real images.

[0006] To achieve the above-mentioned object, the present invention provides a sketch progressive image translation method based on a conditional diffusion model, the sketch progressive image translation method based on a conditional diffusion model comprising:

[0007] Inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0008] Performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector;

[0009] Performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0010] Image translation is performed according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a real sketch image. The conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0011] Optionally, the step of inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch includes:

[0012] Inputting the sketch to be processed into the latent space unit in the grayscale image generation model to obtain a sketch variable of a preset dimension, and obtaining a latent coding sketch according to the sketch variable of the preset dimension;

[0013] Inputting the latent coding sketch into the conditional diffusion noise unit in the grayscale image generation model, and obtaining a noisy grayscale image through the noise addition formula corresponding to the conditional diffusion noise unit;

[0014] The noise adding formula is:

[0015]

[0016]

[0017] Where x0′ is the sketch of the latent code, y′ is the grayscale image of the latent code, and x t ′ is a noise sketch with t-step noise added, t is the current number of diffusion steps, is a scalar parameter, a t =1-β t , β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution;

[0018] The noise sketch is denoised and transformed by a denoising diffusion method according to the latently coded grayscale image to obtain a sketch grayscale image.

[0019] Optionally, the step of performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector includes:

[0020] Perform feature extraction on the sketch to be processed and the sketch grayscale image respectively by a feature extraction function to obtain a sketch vector and a grayscale image vector;

[0021] The feature extraction function is:

[0022] I lowres =F down (I in )

[0023]

[0024]

[0025] v cls =(v sket ,v gary )

[0026] In the formula, I in is a sketch to be processed and a sketch grayscale image; F down is a down-sampling block, I lowres is a feature spectrum after dimension reduction, F attn is a function of internally alternating a multi-layer perception and an attention mechanism to process features, is a feature processed by an attention mechanism, F gap is a global average pooling function, v out is an output vector, v sket is a sketch vector, v gary is a grayscale image vector, v cls is a vector set of classification.

[0027] Optionally, the step of fusing the sketch vector and the grayscale image vector to obtain image fusion feature information comprises:

[0028] fusing the sketch vector and the grayscale image vector by an adaptive weight fusion formula to obtain image fusion feature information;

[0029] The adaptive weight fusion formula is:

[0030] v ff =α1*v sket +α2*v gray *F avg

[0031]

[0032] In the formula, v ff is image fusion feature information, F avg is an average pooling weight factor, a1 is a weight of sketch features, a2 is a weight of grayscale image features, w i and w j are weight matrices.

[0033] Optionally, the step of performing image translation by a conditional diffusion model according to the image fusion feature information and the sketch grayscale image to obtain a sketch real image comprises:

[0034] Inputting the image fusion feature information and the sketch grayscale image into a forward diffusion unit in a conditional diffusion model, and obtaining a noise sketch grayscale image through a conditional diffusion forward formula in the forward diffusion unit;

[0035] The noise sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the real sketch image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

[0036] Optionally, the conditional diffusion forward formula is:

[0037]

[0038] Where x0 is the real image of the potential code, x t is the grayscale image of the noise sketch with t-step noise added, β t and α t is the predefined noise scheduling parameter, ∈ is Gaussian noise, and I is the identity matrix.

[0039] Optionally, the conditional diffusion inverse formula is:

[0040]

[0041]

[0042] Where μ(x t ,y) is the mean, is the covariance matrix, ∈ θ To estimate the noise, x t-1 is the image generated at time step t-1, and L is the loss function.

[0043] In addition, to achieve the above-mentioned purpose, the present invention further proposes a sketch progressive image translation system based on a conditional diffusion model, wherein the sketch progressive image translation system based on the conditional diffusion model comprises:

[0044] a processing module, configured to input a sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0045] An extraction module, configured to perform feature extraction on the sketch to be processed and the sketch grayscale image, respectively, to obtain a sketch vector and a grayscale image vector;

[0046] A fusion module, configured to fuse the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0047] The translation module is configured to perform image translation on the image fusion feature information and the sketch grayscale image based on a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model comprises a forward diffusion unit and a reverse diffusion unit.

[0048] In addition, to achieve the above object, the application further provides a sketch progressive image translation device based on a conditional diffusion model, which comprises a memory, a processor and a sketch progressive image translation program based on a conditional diffusion model stored in the memory and executable on the processor, wherein the sketch progressive image translation program based on a conditional diffusion model is configured to implement the steps of the sketch progressive image translation method based on a conditional diffusion model as described above.

[0049] In addition, to achieve the above object, the application further provides a storage medium, wherein the storage medium stores a sketch progressive image translation program based on a conditional diffusion model, and the sketch progressive image translation program based on a conditional diffusion model implements the steps of the sketch progressive image translation method based on a conditional diffusion model as described above when executed by a processor.

[0050] Firstly, the sketch to be processed is input into a grayscale image generation model to obtain a sketch grayscale image, and the grayscale image generation model comprises a latent space unit and a conditional diffusion noise unit. Then, feature extraction is performed on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector. After that, feature fusion is performed on the sketch vector and the grayscale image vector to obtain image fusion feature information. Finally, image translation is performed on the image fusion feature information and the sketch grayscale image based on a conditional diffusion model to obtain a sketch real image, and the conditional diffusion model comprises a forward diffusion unit and a reverse diffusion unit. In the application, the method of generating a grayscale image from a sketch and then generating a real image from the grayscale image makes the model have strong generalization ability. At the same time, the model can effectively identify and ignore the noise and stains on the sketch, greatly improving the robustness of the model. Moreover, the combination of the sketch features and the generated grayscale image and then the addition to the translation process from the grayscale image to the real image can excellently maintain the accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a structural schematic diagram of a sketch progressive image translation device based on a conditional diffusion model of a hardware running environment involved in an embodiment scheme of the application;

[0052] Figure 2 is a flowchart of a first embodiment of a sketch progressive image translation method based on a conditional diffusion model of the application;

[0053] Figure 3 This is a model architecture diagram of the clothing sketch progressive image translation method of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention;

[0054] Figure 4 This is a structural block diagram of the first embodiment of the sketch progressive image translation system based on the conditional diffusion model of the present invention.

[0055] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a sketch progressive image translation device based on a conditional diffusion model in the hardware operating environment involved in an embodiment of the present invention.

[0058] like Figure 1 As shown, the sketch progressive image translation device based on the conditional diffusion model may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage. The memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0059] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the sketch progressive image translation device based on the conditional diffusion model, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0060] like Figure 1As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a sketch progressive image translation program based on a conditional diffusion model.

[0061] exist Figure 1 In the sketch progressive image translation device based on the conditional diffusion model shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the sketch progressive image translation device based on the conditional diffusion model of the present invention can be set in the sketch progressive image translation device based on the conditional diffusion model, and the sketch progressive image translation device based on the conditional diffusion model calls the sketch progressive image translation program based on the conditional diffusion model stored in the memory 1005 through the processor 1001, and executes the sketch progressive image translation method based on the conditional diffusion model provided by the embodiment of the present invention.

[0062] The embodiment of the present invention provides a sketch progressive image translation method based on a conditional diffusion model, referring to Figure 2 , Figure 2 2 is a flow chart of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention.

[0063] In this embodiment, the sketch progressive image translation method based on the conditional diffusion model includes the following steps:

[0064] Step S10: Inputting the sketch to be processed into a grayscale image generation model to obtain a sketch grayscale image, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit.

[0065] It is easy to understand that the execution entity of this embodiment can be a sketch progressive image translation system based on a conditional diffusion model with functions such as data processing, network communication and program running, or it can be other computer devices with similar functions, etc. This embodiment is not limited to this.

[0066] It should be noted that the sketch to be processed can be a clothing sketch, or a furniture sketch or a decoration sketch, etc. Figure 3 , Figure 3 This is a model architecture diagram of the clothing sketch progressive image translation method of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention. The clothing sketch is used for image translation to finally obtain a real clothing image.

[0067] It should be understood that the grayscale image generation model includes a latent space unit and a conditional diffusion noise unit. The latent space unit is used to reduce the complexity of the clothing image to a manageable level while retaining the important features in the clothing image, so that the clothing image can be more easily processed and analyzed; the conditional diffusion unit is used to add different degrees of noise to the image and then reverse denoise to achieve image-to-image conversion.

[0068] The latent space unit feeds the sketch into a pre-trained latent encoder D, which is essentially an RGB three-channel image. After passing through encoder D, the sketch variable of a preset dimension is generated, which is then further discretized and encoded. Specifically, a codebook of discrete values ​​is pre-generated. At each code position, the nearest code is found in the codebook to generate a variable of the same dimension, i.e., the sketch variable of the preset dimension, thereby reducing the image size.

[0069] Furthermore, the sketch to be processed is input into the grayscale image generation model, and the processing method for obtaining the sketch grayscale image is as follows: the sketch to be processed is input into the latent space unit in the grayscale image generation model to obtain the sketch variable of preset dimension, and the latent coding sketch is obtained according to the sketch variable of preset dimension; the latent coding sketch is input into the conditional diffusion noise unit in the grayscale image generation model, and the noise grayscale image is obtained by the noise addition formula corresponding to the conditional diffusion noise unit; the noise sketch is denoised and converted according to the latent coding grayscale image by the denoising diffusion method to obtain the sketch grayscale image.

[0070] The formula for adding noise is:

[0071]

[0072] Where x0′ is the sketch of the latent code, y′ is the grayscale image of the latent code, and x t ′ is a noise sketch with t-step noise added, t is the current number of diffusion steps, is a scalar parameter, a t =1-β t , β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution;

[0073] In the specific implementation, after obtaining the noise image x t 'Afterwards, based on the main idea of ​​the denoising diffusion method, the inverse process of this method aims to predict the state at the next moment based on the state at the current moment. The posterior distribution formula is as follows:

[0074]

[0075] where p(x t-1 '|x t ',y') is the posterior distribution, μ(x t ',y') is the parameter prediction average of the noise, which needs to be learned by a neural network with parameter θ based on the maximum likelihood criterion. is the covariance matrix.

[0076] By reparameterization method, the mean μ θ Re-express as Re-express as x t ',y and estimated noise∈ θ A linear combination of:

[0077]

[0078] The variance σ is obtained t and mean μ θ Afterwards, the posterior distribution p(x t-1 '|x t ',y') the denoising formula can be obtained as shown below.

[0079]

[0080] Since the network training process is repeated from time t-1 to 0, x t 'Restore to x t-1 ' process, after denoising, the conditional diffusion noise module successfully converts the noise sketch into a grayscale image, namely the sketch grayscale image.

[0081] Step S20: performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector.

[0082] It should be noted that after obtaining the grayscale image, since this example method uses progressive image translation, some of the original features of the sketch may be lost in the process of converting the sketch to the grayscale image. If the grayscale image is directly used as input to generate a real color image, the lack of necessary sketch details may lead to a decrease in quality or even substandard results. Therefore, a feature extraction module is used.

[0083] The feature extraction module first uses the downsampling block to reduce the resolution of the image features of the input image (i.e., the sketch to be processed and the sketch grayscale image). Then, it alternately uses the multi-layer perceptron and attention layer to use the reduced features to generate feature images. Finally, global average pooling is used on the last feature map to obtain the vector used by the classifier.

[0084] Furthermore, feature extraction is performed on the sketch to be processed and the sketch grayscale image respectively to obtain the sketch vector and the grayscale image vector. The processing method is to extract features from the sketch to be processed and the sketch grayscale image respectively through a feature extraction function to obtain the sketch vector and the grayscale image vector;

[0085] The feature extraction function is:

[0086] I lowres =F down (I in )

[0087]

[0088]

[0089] v cls =(v sket ,v gary )

[0090] Where, I in is the sketch to be processed and the sketch grayscale image; F down is the downsampling block, I lowres is the feature map after dimensionality reduction, F attn It is a function that uses multi-layer perceptron and attention mechanism to process features alternately. is the feature processed by the attention mechanism, F gap is the global average pooling function, v out is the output vector, v sket is the sketch vector, v gary is the grayscale image vector, v cls A collection of classified vectors.

[0091] Step S30: performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information.

[0092] It's also important to note that after obtaining the sketch vector (i.e., the vector representing the sketch's features) and the grayscale image vector (i.e., the feature vector representing the sketch's grayscale image), the feature fusion module not only accepts the grayscale image as input but also fuses the sketch features, feeding the fused image features into the next stage of learning. The feature fusion module fuses the sketch features with the generated grayscale image features using adaptive weights.

[0093] In a specific implementation, the sketch vector and the grayscale image vector are subjected to feature fusion to obtain image fusion feature information. The processing method is to perform feature fusion on the sketch vector and the grayscale image vector through an adaptive weight fusion formula to obtain image fusion feature information.

[0094] The adaptive weight fusion formula is:

[0095] v ff =α1*v sket +α2*v gray *F avg

[0096]

[0097] Where, v ff is the image fusion feature information, F avg is the average pooling weight factor, a1 is the weight of the sketch feature, a2 is the weight of the grayscale feature, and w i and w j is the weight matrix.

[0098] Step S40: performing image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a real sketch image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0099] Furthermore, the image fusion feature information and the sketch grayscale image are input into the forward diffusion unit in the conditional diffusion model, and the noise sketch grayscale image is obtained through the conditional diffusion forward formula in the forward diffusion unit; the noise sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the sketch real image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

[0100] In the specific implementation, after obtaining the grayscale image (i.e., the grayscale image of the sketch) and the fused features (i.e., the image fusion feature information), a conditional diffusion model guided by the clothing fusion features is used to simultaneously receive the grayscale image and the fused features for training. The forward process of conditional diffusion is still to add noise to the data until it approaches a Gaussian distribution.

[0101]

[0102] An important feature of the diffusion process is that we can directly use the original data x0 to predict the x at any step. t Sampling is performed, and the re-parameterization technique can be used to obtain:

[0103]

[0104] After inverse reparameterization, we can get:

[0105]

[0106] Where x0 is the real image of the potential code, x t is the grayscale image of the noise sketch with t-step noise added, β t and α tis the predefined noise scheduling parameter, ∈ is Gaussian noise, and I is the identity matrix. and is the combination coefficient, q(x t |x0) is the forward transition probability,

[0107] Furthermore, unlike existing diffusion-based image-to-image translation methods, by introducing an additional input v ff , to modify the posterior distribution p(x t-1 |x t ,y) The model simultaneously receives the previously generated grayscale image y and the mixed feature v ff To guide the diffusion process, it can better understand the context information and provide more precise guidance in the reverse diffusion process. The posterior distribution formula is as follows:

[0108]

[0109]

[0110] Where μ(x t ,y,v ff ) is the mean, σ t is the variance, and the corresponding training loss function is as follows.

[0111]

[0112] Where ∈ θ (x t ,t,y,v ff ) represents the noise of the model prediction, is the gradient of the data distribution. By minimizing this loss, the model’s output can be made closer to the gradient of the real data.

[0113]

[0114] Since the network training process is repeated from time t-1 to 0, x t Restore to x t-1 After denoising, the conditional diffusion model successfully converts the grayscale image into a real image.

[0115] In this embodiment, the sketch to be processed is first input into a grayscale image generation model to obtain a sketch grayscale image. The grayscale image generation model includes a latent space unit and a conditional diffusion noise unit. Feature extraction is then performed on the sketch to be processed and the sketch grayscale image, respectively, to obtain a sketch vector and a grayscale image vector. The sketch vector and the grayscale image vector are then feature-fused to obtain image fusion feature information. Finally, image translation is performed based on the image fusion feature information and the sketch grayscale image using a conditional diffusion model to obtain the sketch real image. The conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit. In this embodiment, by first generating a grayscale image from the sketch and then generating a real image from the grayscale image, the model has strong generalization capabilities. At the same time, the model can effectively identify and ignore noise and stains that appear on the sketch, greatly enhancing the robustness of the model. Combining the sketch features with the generated grayscale image and adding them to the translation process from the grayscale image to the real image can effectively maintain an accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation.

[0116] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the sketch progressive image translation system based on the conditional diffusion model of the present invention.

[0117] like Figure 4 As shown, the sketch progressive image translation system based on the conditional diffusion model proposed in the embodiment of the present invention includes:

[0118] Processing module 4001, configured to input a sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0119] Extraction module 4002, used to perform feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector;

[0120] A fusion module 4003 is used to fuse the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0121] The translation module 4004 is used to perform image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a real sketch image. The conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0122] In this embodiment, the sketch to be processed is first input into a grayscale image generation model to obtain a sketch grayscale image. The grayscale image generation model includes a latent space unit and a conditional diffusion noise unit. Feature extraction is then performed on the sketch to be processed and the sketch grayscale image, respectively, to obtain a sketch vector and a grayscale image vector. The sketch vector and the grayscale image vector are then feature-fused to obtain image fusion feature information. Finally, image translation is performed based on the image fusion feature information and the sketch grayscale image using a conditional diffusion model to obtain the sketch real image. The conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit. In this embodiment, by first generating a grayscale image from the sketch and then generating a real image from the grayscale image, the model has strong generalization capabilities. At the same time, the model can effectively identify and ignore noise and stains that appear on the sketch, greatly enhancing the robustness of the model. Combining the sketch features with the generated grayscale image and adding them to the translation process from the grayscale image to the real image can effectively maintain an accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation.

[0123] Other embodiments or specific implementations of the sketch progressive image translation system based on the conditional diffusion model of the present invention can refer to the above-mentioned method embodiments and will not be repeated here.

[0124] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0125] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0127] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A sketch progressive image translation method based on a conditional diffusion model, characterized in that: The sketch progressive image translation method based on the conditional diffusion model comprises the following steps: Inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit; Performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector; Performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information; Performing image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a true sketch image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit; The step of respectively extracting features from the sketch to be processed and the sketch grayscale image to obtain a sketch vector and a grayscale image vector comprises: Perform feature extraction on the sketch to be processed and the sketch grayscale image respectively by a feature extraction function to obtain a sketch vector and a grayscale image vector; The feature extraction function is: I lowres =F down (I in ) v cls =(v sket ,v gary ) Where, I in is the sketch to be processed and the sketch grayscale image; F down is the downsampling block, I lowres is the feature map after dimensionality reduction, F attn It is a function that uses multi-layer perceptron and attention mechanism to process features alternately. is the feature processed by the attention mechanism, F gap is the global average pooling function, v out is the output vector, v sket is the sketch vector, v gary is the grayscale image vector, v cls is a set of classified vectors; The conditional diffusion inverse formula in the inverse diffusion unit is: In the formula, μ(x t ,y,v ff ) is the mean, is the covariance matrix, ∈ θ To estimate the noise, q(x t |x0) is the conditional probability distribution in the forward diffusion process, For q(x t |x0)About x t The true gradient, x t-1 is the image generated at time step t-1, x t is the image generated at time step t, x0 is the sketch grayscale image, t is the current diffusion step number, y is the real image of the potential code, L is the loss function, σ t z is the random noise added during the reverse diffusion process, is a normal distribution, β t and α t is the predefined noise scheduling parameter, v ff It is the image fusion feature information.

2. The method according to claim 1, wherein The step of inputting the sketch to be processed into the grayscale image generation model to obtain the sketch grayscale image includes: Inputting the sketch to be processed into the latent space unit in the grayscale image generation model to obtain a sketch variable of a preset dimension, and obtaining a latent coding sketch according to the sketch variable of the preset dimension; Inputting the latent coding sketch into the conditional diffusion noise unit in the grayscale image generation model, and obtaining a noisy grayscale image through the noise addition formula corresponding to the conditional diffusion noise unit; The noise adding formula is: Where x0′ is the original image to which noise needs to be added after latent coding, y′ is the grayscale image of the latent coding, and x t ′ is the noise image with t-step noise added, t is the current number of diffusion steps, is a scalar parameter, β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution; The noise sketch is denoised and transformed by a denoising diffusion method according to the latently coded grayscale image to obtain a sketch grayscale image.

3. The method according to claim 1, wherein The step of performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information includes: Performing feature fusion on the sketch vector and the grayscale image vector using an adaptive weight fusion formula to obtain image fusion feature information; The adaptive weight fusion formula is: in ff =α1*v sket +α2*v gray *F avg Where, v ff is the image fusion feature information, F avg is the average pooling weight factor, a1 is the weight of the sketch feature, a2 is the weight of the grayscale feature, and w i and w j is the weight matrix.

4. The method according to claim 3, wherein The step of performing image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a true sketch image includes: Inputting the image fusion feature information and the sketch grayscale image into a forward diffusion unit in a conditional diffusion model, and obtaining a noise sketch grayscale image through a conditional diffusion forward formula in the forward diffusion unit; The noise sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the real sketch image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

5. The method according to claim 4, wherein The conditional diffusion forward formula is: Where x0 is the sketch grayscale image, y is the real image of the latent code, and x t is the grayscale image of the noise sketch with t-step noise added, x t-1 is the grayscale image of the noise sketch with t-1 steps of noise added, β t and α t is the predefined noise scheduling parameter, is a normal distribution, α s is the noise scheduling parameter at the sth step, ∈ is Gaussian noise, and I is the identity matrix.

6. A sketch progressive image translation system based on a conditional diffusion model, characterized in that: The sketch progressive image translation system based on the conditional diffusion model includes: a processing module, configured to input a sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit; An extraction module, configured to perform feature extraction on the sketch to be processed and the sketch grayscale image, respectively, to obtain a sketch vector and a grayscale image vector; A fusion module, configured to fuse the sketch vector and the grayscale image vector to obtain image fusion feature information; A translation module, configured to perform image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a true sketch image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit; The sketch progressive image translation system based on the conditional diffusion model implements the steps of the sketch progressive image translation method based on the conditional diffusion model as claimed in any one of claims 1 to 5.

Images