Sketch progressive image translation method and system based on conditional diffusion model

Abstract

The invention discloses a sketch progressive image translation method and system based on a conditional diffusion model, and the method comprises the steps: inputting a to-be-processed sketch into a grayscale image generation model, and obtaining a sketch grayscale image, the grayscale image generation model comprising a potential space unit and a conditional diffusion noise unit; respectively carrying out feature extraction on the sketch to be processed and the sketch gray-scale image to obtain a sketch vector and a gray-scale image vector; performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information; image translation is carried out through a conditional diffusion model according to the image fusion feature information and the sketch grayscale image, a real sketch image is obtained, and the conditional diffusion model comprises a forward diffusion unit and a reverse diffusion unit. According to the method, sketch features and the generated grey-scale map are combined and then added into the translation process from the grey-scale map to the real image, so that accurate understanding of the overall structure and local features of the sketch can be excellently kept, and the image translation quality is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a sketch progressive image translation method and system based on a conditional diffusion model. Background Art

[0002] With the continuous development of computer vision and natural language processing, image translation has attracted widespread attention as an important research task. Image translation aims to achieve image conversion between different modalities, such as converting RGB images to thermal images or converting painting sketches to corresponding real images. This task is of great significance for realizing the conversion of cross-modal information and can provide solutions for many practical application scenarios, such as virtual reality, auxiliary medical diagnosis and artistic creation.

[0003] In recent years, methods based on GAN and diffusion have made significant progress in the field of image translation. However, GAN-based models such as CycleGAN and SketchyGAN have disadvantages such as gradient vanishing and mode collapse, and poor generalization ability. Diffusion-based models such as UNIT-DDPM are prone to pixels being incorrectly colored. Image translation tasks, especially those involving translation from sketches to normal images, still face many challenges.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a sketch progressive image translation method and system based on a conditional diffusion model, aiming to solve the technical problem of how to accurately translate a sketch into a real image.

[0006] To achieve the above object, the present invention provides a sketch progressive image translation method based on a conditional diffusion model, the sketch progressive image translation method based on a conditional diffusion model comprising:

[0007] Inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0008] Performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector;

[0009] Performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0010] Image translation is performed according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0011] Optionally, the step of inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch includes:

[0012] Inputting the sketch to be processed into the latent space unit in the grayscale image generation model to obtain a sketch variable of a preset dimension, and obtaining a latent coding sketch according to the sketch variable of the preset dimension;

[0013] Inputting the latent coding sketch into the conditional diffusion noise unit in the grayscale image generation model, and obtaining a noisy grayscale image through a noise adding formula corresponding to the conditional diffusion noise unit;

[0014] The noise adding formula is:

[0015]

[0016] In the formula, x 0 ′ is the sketch of the latent code, y′ is the grayscale image of the latent code, and x t ′ is a noise sketch with t-step noise added, t is the current diffusion step number, is a scalar parameter, a t =1-β t , β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution;

[0017] The noisy sketch is denoised and transformed according to the potentially encoded grayscale image by a denoising diffusion method to obtain a sketch grayscale image.

[0018] Optionally, the step of respectively extracting features from the sketch to be processed and the sketch grayscale image to obtain a sketch vector and a grayscale image vector comprises:

[0019] Extract features from the sketch to be processed and the sketch grayscale image respectively through a feature extraction function to obtain a sketch vector and a grayscale image vector;

[0020] The feature extraction function is:

[0021] I lowres =F down (I in )

[0022] I attnfeat =Fattn (I lowres )

[0023] v out =F gap (I attnfeat )

[0024] v cls =(v sket ,v gary )

[0025] In the formula, I in is the sketch to be processed and the sketch grayscale image; F down is the downsampling block, I lowres is the feature map after dimensionality reduction, F attn It is a function that uses multi-layer perceptron and attention mechanism to process features alternately. attnfeat is the feature processed by the attention mechanism, F gap is the global average pooling function, v out is the output vector, v sket is the sketch vector, v gary is the grayscale image vector, v cls A collection of vectors for classification.

[0026] Optionally, the step of performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information includes:

[0027] Performing feature fusion on the sketch vector and the grayscale image vector through an adaptive weight fusion formula to obtain image fusion feature information;

[0028] The adaptive weight fusion formula is:

[0029] v ff =α 1 *v sket +α 2 *v gray *F avg

[0030]

[0031] In the formula, v ff is the image fusion feature information, F aVg is the average pooling weight factor, a 1 is the weight of the sketch feature, a 2 is the weight of the grayscale image feature, w i and w j is the weight matrix.

[0032] Optionally, the step of performing image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image includes:

[0033] Inputting the image fusion feature information and the sketch grayscale image into a forward diffusion unit in a conditional diffusion model, and obtaining a noise sketch grayscale image through a conditional diffusion forward formula in the forward diffusion unit;

[0034] The noise sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the real sketch image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

[0035] Optionally, the conditional diffusion forward formula is:

[0036]

[0037] α t :=1-β t ,

[0038]

[0039] In the formula, x 0 is the real image of the latent code, x t is the grayscale image of the noisy sketch with t-step noise added, β t and α t is the predefined noise scheduling parameter, ∈ is Gaussian noise, and I is the unit matrix.

[0040] Optionally, the conditional diffusion inverse formula is:

[0041]

[0042] Where μ(x t ,y) is the mean, is the covariance matrix, ∈ θ To estimate the noise, x t-1 is the image generated at time step t-1, and L is the loss function.

[0043] In addition, to achieve the above purpose, the present invention also proposes a sketch progressive image translation system based on a conditional diffusion model, and the sketch progressive image translation system based on a conditional diffusion model includes:

[0044] A processing module, used for inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0045] An extraction module, used for performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector;

[0046] A fusion module, used for fusing the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0047] The translation module is used to perform image translation according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0048] In addition, to achieve the above-mentioned purpose, the present invention also proposes a sketch progressive image translation device based on a conditional diffusion model, the device comprising: a memory, a processor, and a sketch progressive image translation program based on a conditional diffusion model stored in the memory and executable on the processor, the sketch progressive image translation program based on the conditional diffusion model being configured to implement the steps of the sketch progressive image translation method based on the conditional diffusion model as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a sketch progressive image translation program based on a conditional diffusion model is stored. When the sketch progressive image translation program based on the conditional diffusion model is executed by a processor, the steps of the sketch progressive image translation method based on the conditional diffusion model as described above are implemented.

[0050] The present invention first inputs the sketch to be processed into the grayscale image generation model to obtain the sketch grayscale image, the grayscale image generation model includes a latent space unit and a conditional diffusion noise unit, then extracts features from the sketch to be processed and the sketch grayscale image respectively, obtains a sketch vector and a grayscale image vector, then fuses the sketch vector and the grayscale image vector to obtain image fusion feature information, and finally translates the image through the conditional diffusion model according to the image fusion feature information and the sketch grayscale image to obtain the sketch real image, the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit. In the present invention, the method of first generating a grayscale image from a sketch and then generating a real image from the grayscale image enables the model to have a strong generalization ability. At the same time, for the noise and stains appearing on the sketch, the model can effectively identify and ignore these noise points, which greatly enhances the robustness of the model. Combining the sketch features with the generated grayscale image and adding them to the translation process from the grayscale image to the real image can effectively maintain an accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1It is a structural schematic diagram of a sketch progressive image translation device based on a conditional diffusion model in a hardware operating environment involved in an embodiment of the present invention;

[0052] Figure 2 It is a flow chart of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention;

[0053] Figure 3 A model architecture diagram of a clothing sketch progressive image translation method of the first embodiment of the sketch progressive image translation method based on a conditional diffusion model of the present invention;

[0054] Figure 4 It is a structural block diagram of the first embodiment of the sketch progressive image translation system based on the conditional diffusion model of the present invention.

[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0057] Reference Figure 1 , Figure 1 The schematic diagram of the structure of a sketch progressive image translation device based on a conditional diffusion model in the hardware operating environment involved in the embodiment of the present invention.

[0058] like Figure 1 As shown, the sketch progressive image translation device based on the conditional diffusion model may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-VolatileMemory, NVM), such as a disk storage. The memory 1005 may also be a storage system independent of the aforementioned processor 1001.

[0059] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the sketch progressive image translation device based on the conditional diffusion model, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0060] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a sketch progressive image translation program based on a conditional diffusion model.

[0061] exist Figure 1 In the sketch progressive image translation device based on the conditional diffusion model shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the sketch progressive image translation device based on the conditional diffusion model of the present invention can be set in the sketch progressive image translation device based on the conditional diffusion model, and the sketch progressive image translation device based on the conditional diffusion model calls the sketch progressive image translation program based on the conditional diffusion model stored in the memory 1005 through the processor 1001, and executes the sketch progressive image translation method based on the conditional diffusion model provided by the embodiment of the present invention.

[0062] The embodiment of the present invention provides a sketch progressive image translation method based on a conditional diffusion model. Figure 2 , Figure 2 It is a flowchart diagram of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention.

[0063] In this embodiment, the sketch progressive image translation method based on the conditional diffusion model includes the following steps:

[0064] Step S10: Inputting the sketch to be processed into a grayscale image generation model to obtain a sketch grayscale image, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit.

[0065] It is easy to understand that the execution subject of this embodiment can be a sketch progressive image translation system based on a conditional diffusion model with functions such as data processing, network communication and program running, or it can be other computer devices with similar functions, etc., and this embodiment is not limited.

[0066] It should be noted that the sketch to be processed can be a clothing sketch, or a furniture sketch or a decoration sketch, etc. Figure 3 , Figure 3This is a model architecture diagram of a clothing sketch progressive image translation method of the first embodiment of the sketch progressive image translation method based on the conditional diffusion model of the present invention. The clothing sketch is used for image translation to finally obtain a real clothing image.

[0067] It should be understood that the grayscale image generation model includes a latent space unit and a conditional diffusion noise unit. The latent space unit is used to reduce the complexity of the clothing image to a manageable level while retaining the important features in the clothing image, so that the clothing image can be more easily processed and analyzed; the conditional diffusion unit is used to add different degrees of noise to the image, and then reverse denoise to achieve image-to-image conversion.

[0068] The latent space unit inputs the sketch into the pre-trained latent encoder D, that is, the RGB three-channel image. After passing through the encoder D, the sketch variable of the preset dimension is obtained, and then further discretized encoding is performed. The specific method is: a codebook of discrete values ​​is generated in advance, and the code closest to it is found in each encoding position, and a variable with the same dimension, that is, a sketch variable of the preset dimension is generated to achieve the purpose of reducing the image size.

[0069] Furthermore, the sketch to be processed is input into the grayscale image generation model, and the processing method for obtaining the sketch grayscale image is as follows: the sketch to be processed is input into the latent space unit in the grayscale image generation model to obtain the sketch variables of preset dimensions, and the latent coding sketch is obtained according to the sketch variables of preset dimensions; the latent coding sketch is input into the conditional diffusion noise unit in the grayscale image generation model, and the noisy grayscale image is obtained by the noise addition formula corresponding to the conditional diffusion noise unit; the noisy sketch is denoised and converted according to the latent coded grayscale image by the denoising diffusion method, so as to obtain the sketch grayscale image.

[0070] The formula for adding noise is:

[0071]

[0072] In the formula, x 0 ′ is the sketch of the latent code, y′ is the grayscale image of the latent code, and x t ′ is a noise sketch with t-step noise added, t is the current diffusion step number, is a scalar parameter, a t =1-β t , β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution;

[0073] In the specific implementation, after obtaining the noise-added image xt 'After that, based on the main idea of ​​the denoising diffusion method, the inverse process of this method aims to predict the state at the next moment based on the state at the current moment. The posterior distribution formula is as follows:

[0074]

[0075] Where p(x t-1 '|x t ',y') is the posterior distribution, μ(x t ',y') is the parameter prediction average of the noise, which needs to be learned by a neural network with parameter θ based on the maximum likelihood criterion. is the covariance matrix.

[0076] By reparameterization method, the mean μ θ Re-express as Re-express as x t ',y and estimated noise∈ θ A linear combination of:

[0077]

[0078] The variance σ is obtained t With mean μ θ Afterwards, the posterior distribution p(x t-1 '|x t ',y') the denoising formula can be obtained as shown below.

[0079]

[0080] Since the network training process is repeated step by step from time t-1 to 0 t 'Restore to x t-1 ' process, after denoising, the conditional diffusion noise module successfully converts the noise sketch into a grayscale image, namely the sketch grayscale image.

[0081] Step S20: extracting features from the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector.

[0082] It should be noted that after obtaining the grayscale image, since the method of this example adopts progressive image translation, some original features of the sketch will be lost in the process from the sketch to the grayscale image. If the grayscale image is directly used as input to generate a colored real image, the quality of the generated image may be reduced or not meet expectations due to the lack of necessary sketch details. Therefore, the feature extraction module is used.

[0083] The feature extraction module first uses the downsampling block to reduce the resolution of the image features of the input image (i.e., the sketch to be processed and the sketch grayscale image), and then alternately uses the multi-layer perceptron and the attention layer to use the reduced-dimensional features to generate feature images, and then uses global average pooling on the last feature map to obtain the vector used by the classifier.

[0084] Further, feature extraction is performed on the sketch to be processed and the sketch grayscale image respectively, and the processing method of obtaining the sketch vector and the grayscale image vector is to extract features of the sketch to be processed and the sketch grayscale image respectively through a feature extraction function to obtain the sketch vector and the grayscale image vector;

[0085] The feature extraction function is:

[0086] I lowres =F down (I in )

[0087] I attnfeat =F attn (I lowres )

[0088] v out =F gap (I attnfeat )

[0089] v cls =(v sket, v gary )

[0090] In the formula, I in is the sketch to be processed and the sketch grayscale image; F down is the downsampling block, I lowres is the feature map after dimensionality reduction, F attn It is a function that uses multi-layer perceptron and attention mechanism to process features alternately. attnfeat is the feature processed by the attention mechanism, F gap is the global average pooling function, v out is the output vector, v sket is the sketch vector, v gary is the grayscale image vector, v cls A collection of vectors for classification.

[0091] Step S30: performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information.

[0092] It should also be noted that after obtaining the sketch vector (i.e., the vector about the sketch feature) and the grayscale image vector (i.e., the feature vector about the sketch grayscale image), the feature fusion module is used to not only receive the grayscale image as input, but also fuse the sketch features and put the fused image features into the next stage of learning. The feature fusion module fuses the sketch features with the generated grayscale image features according to the adaptive weights.

[0093] In a specific implementation, the sketch vector and the grayscale image vector are subjected to feature fusion to obtain image fusion feature information. The processing method is to perform feature fusion on the sketch vector and the grayscale image vector through an adaptive weight fusion formula to obtain image fusion feature information;

[0094] The adaptive weight fusion formula is:

[0095] v ff =α 1 *v sket +α 2 *v gray *F avg

[0096]

[0097] In the formula, v ff is the image fusion feature information, F avg is the average pooling weight factor, a 1 is the weight of the sketch feature, a 2 is the weight of the grayscale image feature, w i and w j is the weight matrix.

[0098] Step S40: performing image translation according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0099] Furthermore, the image fusion feature information and the sketch grayscale image are input into the forward diffusion unit in the conditional diffusion model, and the noisy sketch grayscale image is obtained through the conditional diffusion forward formula in the forward diffusion unit; the noisy sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the sketch real image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

[0100] In the specific implementation, after obtaining the grayscale image (i.e. the sketch grayscale image) and the fusion feature (i.e. the image fusion feature information), the conditional diffusion model guided by the clothing fusion feature is used to simultaneously receive the grayscale image and the fusion feature for training. The forward process of conditional diffusion is still to add noise to the data until it tends to Gaussian distribution.

[0101]

[0102] An important property of the diffusion process is that we can directly 0 For any step x t Sampling is performed, and the re-parameterization technique can be used to obtain:

[0103] α t :=1-β t ,

[0104]

[0105] After anti-reparameterization, we can get:

[0106]

[0107] In the formula, x 0 is the real image of the latent code, x t is the grayscale image of the noisy sketch with t-step noise added, β t and α t is the predefined noise scheduling parameter, ∈ is Gaussian noise, and I is the unit matrix. and is the combination coefficient, q(x t |x 0 ) is the forward transition probability,

[0108] Furthermore, unlike existing diffusion-based image-to-image translation methods, by introducing an additional input v in the reverse process ff , to modify the posterior distribution p(x t-1 |x t ,y) The model simultaneously receives the previously generated grayscale image y and the mixed feature v ff To guide the diffusion process, so that it can better understand the context information and provide more precise guidance in the reverse diffusion process. The posterior distribution formula is as follows:

[0109]

[0110] Where μ(x t ,y,v ff ) is the mean, σ t is the variance, and the corresponding training loss function is as follows.

[0111]

[0112] Where ∈ θ (x t ,t,y,v ff ) represents the noise predicted by the model, is the gradient of the data distribution. By minimizing this loss, the output of the model can bring it closer to the gradient of the true data.

[0113]

[0114] Since the network training process is repeated step by step from time t-1 to 0 t Restore to x t-1 After denoising, the conditional diffusion model successfully converts the grayscale image into a real image.

[0115] In this embodiment, the sketch to be processed is first input into the grayscale image generation model to obtain the sketch grayscale image, the grayscale image generation model includes a latent space unit and a conditional diffusion noise unit, and then the sketch to be processed and the sketch grayscale image are respectively extracted to obtain a sketch vector and a grayscale image vector, and then the sketch vector and the grayscale image vector are feature fused to obtain image fusion feature information, and finally the image is translated through the conditional diffusion model according to the image fusion feature information and the sketch grayscale image to obtain the sketch real image, and the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit. In this embodiment, the method of first generating a grayscale image from a sketch and then generating a real image from the grayscale image makes the model have a strong generalization ability. At the same time, for the noise and stains appearing on the sketch, the model can effectively identify and ignore these noise points, which greatly enhances the robustness of the model. Combining the sketch features with the generated grayscale image and adding them to the translation process from the grayscale image to the real image can effectively maintain an accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation.

[0116] Reference Figure 4 , Figure 4 It is a structural block diagram of the first embodiment of the sketch progressive image translation system based on the conditional diffusion model of the present invention.

[0117] like Figure 4 As shown, the sketch progressive image translation system based on the conditional diffusion model proposed in the embodiment of the present invention includes:

[0118] A processing module 4001 is used to input the sketch to be processed into a grayscale image generation model to obtain a sketch grayscale image, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit;

[0119] An extraction module 4002 is used to extract features from the sketch to be processed and the sketch grayscale image to obtain a sketch vector and a grayscale image vector;

[0120] A fusion module 4003 is used to fuse the sketch vector and the grayscale image vector to obtain image fusion feature information;

[0121] The translation module 4004 is used to perform image translation according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

[0122] In this embodiment, the sketch to be processed is first input into the grayscale image generation model to obtain the sketch grayscale image, the grayscale image generation model includes a latent space unit and a conditional diffusion noise unit, and then the sketch to be processed and the sketch grayscale image are respectively extracted to obtain a sketch vector and a grayscale image vector, and then the sketch vector and the grayscale image vector are feature fused to obtain image fusion feature information, and finally the image is translated through the conditional diffusion model according to the image fusion feature information and the sketch grayscale image to obtain the sketch real image, and the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit. In this embodiment, the method of first generating a grayscale image from a sketch and then generating a real image from the grayscale image makes the model have a strong generalization ability. At the same time, for the noise and stains appearing on the sketch, the model can effectively identify and ignore these noise points, which greatly enhances the robustness of the model. Combining the sketch features with the generated grayscale image and adding them to the translation process from the grayscale image to the real image can effectively maintain an accurate understanding of the overall structure and local features of the sketch, improve the quality of image translation, and effectively improve the image quality and diversity based on sketch image translation.

[0123] Other embodiments or specific implementations of the sketch progressive image translation system based on the conditional diffusion model of the present invention can refer to the above-mentioned method embodiments, which will not be described in detail here.

[0124] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0125] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0127] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A sketch progressive image translation method based on a conditional diffusion model, characterized in that: The sketch progressive image translation method based on the conditional diffusion model comprises the following steps: Inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit; Performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector; Performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information; Image translation is performed according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

2. The method according to claim 1, characterized in that The step of inputting the sketch to be processed into the grayscale image generation model to obtain the sketch grayscale image includes: Inputting the sketch to be processed into the latent space unit in the grayscale image generation model to obtain a sketch variable of a preset dimension, and obtaining a latent coding sketch according to the sketch variable of the preset dimension; Inputting the latent coding sketch into the conditional diffusion noise unit in the grayscale image generation model, and obtaining a noisy grayscale image through a noise adding formula corresponding to the conditional diffusion noise unit; The noise adding formula is: In the formula, x0′ is the original image to which noise needs to be added after latent coding, x t ′ is the noise image with t-step noise added, t is the current number of diffusion steps, is a scalar parameter, a t =1-β t , β t is the linear variance table, δ t To control the variance of the noise, z is random noise, is a normal distribution, I is the variance of the normal distribution; The noisy sketch is denoised and transformed according to the potentially encoded grayscale image by a denoising diffusion method to obtain a sketch grayscale image.

3. The method according to claim 2, characterized in that The step of respectively extracting features from the sketch to be processed and the sketch grayscale image to obtain a sketch vector and a grayscale image vector comprises: Extract features from the sketch to be processed and the sketch grayscale image respectively through a feature extraction function to obtain a sketch vector and a grayscale image vector; The feature extraction function is: I lowres =F down (I in ) I attnfeat =F attn (I lowres ) v out =F gap (I attnfeat ) v cls =(v sket ,v gary ) In the formula, I in is the sketch to be processed and the sketch grayscale image; F down is the downsampling block, I lowres is the feature map after dimensionality reduction, F attn It is a function that uses multi-layer perceptron and attention mechanism to process features alternately. attnfeat is the feature processed by the attention mechanism, F gap is the global average pooling function, v out is the output vector, v sket is the sketch vector, v gary is the grayscale image vector, v cls A collection of vectors for classification.

4. The method according to claim 3, characterized in that The step of performing feature fusion on the sketch vector and the grayscale image vector to obtain image fusion feature information comprises: Performing feature fusion on the sketch vector and the grayscale image vector through an adaptive weight fusion formula to obtain image fusion feature information; The adaptive weight fusion formula is: in ff =α1*v sket +α2*v gray *F avg In the formula, v ff is the image fusion feature information, F avg is the average pooling weight factor, a1 is the weight of the sketch feature, a2 is the weight of the grayscale feature, and w i and w j is the weight matrix.

5. The method according to claim 4, characterized in that The step of performing image translation based on the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image comprises: Inputting the image fusion feature information and the sketch grayscale image into a forward diffusion unit in a conditional diffusion model, and obtaining a noise sketch grayscale image through a conditional diffusion forward formula in the forward diffusion unit; The noise sketch grayscale image is input into the reverse diffusion unit in the conditional diffusion model, and the real sketch image is obtained through the conditional diffusion inverse formula in the reverse diffusion unit.

6. The method according to claim 5, characterized in that The conditional diffusion forward formula is: α t :＝1-β t , Where x0 is the sketch grayscale image, y is the real image of the latent code, and x t is the grayscale image of the noisy sketch with t-step noise added, β t and α t is the predefined noise scheduling parameter, ∈ is Gaussian noise, and I is the unit matrix.

7. The method according to claim 6, characterized in that The inverse formula of conditional diffusion is: Where μ(x t ,y) is the mean, is the covariance matrix, ∈ θ To estimate the noise, x t-1 is the image generated at time step t-1, and L is the loss function.

8. A sketch progressive image translation system based on a conditional diffusion model, characterized in that: The sketch progressive image translation system based on the conditional diffusion model includes: A processing module, used for inputting the sketch to be processed into a grayscale image generation model to obtain a grayscale image of the sketch, wherein the grayscale image generation model includes a latent space unit and a conditional diffuse noise unit; An extraction module, used for performing feature extraction on the sketch to be processed and the sketch grayscale image respectively to obtain a sketch vector and a grayscale image vector; A fusion module, used for fusing the sketch vector and the grayscale image vector to obtain image fusion feature information; The translation module is used to perform image translation according to the image fusion feature information and the sketch grayscale image through a conditional diffusion model to obtain a sketch real image, wherein the conditional diffusion model includes a forward diffusion unit and a reverse diffusion unit.

Images