Image processing method and system based on controllable generation, terminal and medium

Abstract

The invention provides an image processing method and system based on controllable generation, a terminal and a medium, and the method comprises the steps: obtaining a condition signal of an image processing task and an information extraction model, and constructing a first loss function; selecting a pre-trained diffusion model; iterating based on a diffusion model and a first loss function, including: M1, sampling a plurality of samples from initial distribution of sequence Monte Carlo; m2, establishing optimization estimation of suggested distribution and unconditional transition probability distribution, and obtaining a weight equation of sequence Monte Carlo; m3, re-sampling to obtain a new sample; and M1-M3 are repeated until a set iteration step number, a final weight equation and a plurality of sampled samples are obtained, and a finally generated image is obtained. In the tasks of text condition image generation, segmentation image condition image generation, face image condition image generation, style image condition image generation and the like, high-quality, diversified and training-free image condition generation can be realized, and the requirement of general processing is met.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image processing method and system based on controllable generation, a terminal and a medium. Background Art

[0002] With the continuous advancement of media technology, diffusion models have shown advantages over existing generative models in generative tasks, such as generative adversarial networks (GANs), variational autoencoders (VAEs), and normalizing flows. With its powerful generation capabilities, diffusion models have promoted their extensive application research in many downstream tasks. Among these applications, conditional generation has become a highly anticipated field because of its ability to perform controllable generation under given conditions. Many conditional diffusion models have been optimized for different conditions, including category labels, textual cues, degraded images, reference images, segmentation maps, and style images.

[0003] Methods for conditional generation using diffusion models are generally divided into training-based methods and training-free methods. Training-based strategies require retraining the model based on given conditions. Although such methods can bring excellent generation performance, they lack flexibility, especially when facing new conditions, retraining or fine-tuning the model is costly and has poor adaptability. In contrast, training-free conditional generation methods aim to overcome the challenges brought by additional training. These methods use off-the-shelf loss functions combined with pre-trained (unconditional or conditional) diffusion models to guide the sampling process. For example, methods such as "FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model" published at the 2023 Conference on Computer Vision and Pattern Recognition (CVPR), and "Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation" and "Universal Guidance for Diffusion Models" at the 2023 International Conference on Machine Learning (ICML) all solve the problem of image generation under uncertain conditions by guiding conditional sampling through implicit priors and conditional loss functions of diffusion models without retraining the model. However, these methods still have shortcomings in consistency and reliability, and the generated images may not fully meet the given conditions. This is because their heuristic approximations to the conditional score function (such as the Laplace approximation) may introduce errors. Summary of the invention

[0004] In view of the problems that the existing technology of training-free conditional generation based on diffusion model cannot generate images that fully meet the given conditions and the generated images have low quality, the present invention provides an image processing method, system, terminal and medium based on controllable generation, which can generate high-quality images that fully meet the given conditions and meet the current needs in the field of controllable image generation.

[0005] According to a first aspect of the present invention, there is provided a controllably generated image processing method, comprising:

[0006] Acquire a conditional signal and an information extraction model of an image processing task, and construct a first loss function based on the conditional signal and the information extraction model;

[0007] Select a pre-trained diffusion model;

[0008] Iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image;

[0009] Wherein: iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image includes:

[0010] M1, samples multiple samples from the initial distribution of Sequential Monte Carlo as a particle filter;

[0011] M2, based on the pre-trained diffusion model and the first loss function, establish an optimized estimate of the suggestion distribution and an unconditional transition probability distribution to obtain a weight equation of a sequential Monte Carlo;

[0012] M3, resampling to obtain new samples based on the weight equation and resampling strategy of the sequence Monte Carlo;

[0013] Repeat the above steps M1-M3 until the set number of iterations is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

[0014] The present invention can realize diverse high-quality image conditional generation without training in tasks such as text conditional image generation, segmentation map conditional image generation, face map conditional image generation and style map conditional image generation, thus meeting the needs of general processing.

[0015] Preferably, the conditional signal y and the information extraction model H of the image processing task are obtained, and are determined according to the task type. The information extraction model H includes: a denoising image information extraction model Conditional Information Extraction Model include:

[0016] -When the image processing task is text conditional image generation: the conditional signal y is a text condition, the information extraction model H is a CLIP model, and the corresponding denoising image information extraction model For CLIP image encoder, conditional information extraction model It is the CLIP text encoder;

[0017] -When the image processing task is segmentation map condition image generation: the conditional signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model Pre-trained MobileNetV3-Large with segmentation head, conditional information extraction model is an identity operation;

[0018] -When the image processing task is to generate a face image condition: the conditional signal y is the face image condition, the information extraction model H is the face ID feature extraction network, and the corresponding denoising image information extraction model For face ID recognition network, conditional information extraction network For the same face ID recognition network;

[0019] -When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoising image information extraction model Get the Gram matrix of features for the CLIP image encoder, conditional information extraction network The Gram matrix of features is obtained for the same CLIP image encoder.

[0020] Preferably, a loss function is constructed based on the conditional signal and the information extraction network, including:

[0021] -When the image processing task is text conditional image generation: the conditional signal y is the text condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the cosine similarity of given y and x0;

[0022] -When the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the sum of the cross entropy losses of each pixel between the predicted segmentation maps of y and x0;

[0023] -When the image processing task is face image conditional image generation: the conditional signal y is a face image condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0;

[0024] -When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

[0025] According to a second aspect of the present invention, a controllably generated image processing system is characterized by comprising:

[0026] Data acquisition module: obtains the conditional signal y and information extraction model H of the image processing task;

[0027] A construction module: constructing a first loss function based on the conditional signal and information extraction model obtained by the data acquisition module, and at the same time, constructing a diffusion model, and training using a second loss function for predicting noise in an image data set to obtain a pre-trained diffusion model;

[0028] Image processing module: iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image;

[0029] Wherein, the image processing module includes:

[0030] Module M1, which samples multiple samples from the initial distribution of Sequential Monte Carlo, acts as a particle filter;

[0031] Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the suggestion distribution and an unconditional transition probability distribution to obtain a weight equation of a sequential Monte Carlo;

[0032] Module M3, based on the weight equation and resampling strategy of the sequential Monte Carlo, resampling to obtain new samples;

[0033] The above modules M1-M3 are executed in sequence until the set number of iteration steps is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

[0034] According to a third aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described, or to run the system described, when executing the program.

[0035] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above, or to run the system described above.

[0036] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0037] The controllable generation-based image processing method and system provided by the embodiment of the present invention realizes controllable generation through a diffusion model based on a training-free method of Sequential Monte Carlo (SMC). Compared with the existing training-free diffusion model method, this method uses loss-guided diffusion sampling to achieve asymptotically accurate generation under various conditions such as text, style, face image, segmentation image, etc., to generate high-quality images that fully meet the given conditions, and meet the current needs in the field of controllable image generation.

[0038] The image processing method and system based on controllable generation provided by the embodiments of the present invention have a plug-and-play flexibility effect compared to the deep neural network method based on supervised learning, do not require additional training and data collection and preprocessing processes, and save computing resources and human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0040] Figure 1 is an overall flow chart of an image processing method based on controllable generation in one embodiment of the present invention;

[0041] Figure 2 An overall flow chart of the controllable generation of a loss-guided diffusion model provided by a preferred embodiment of the present invention.

[0042] Figure 3 A detailed flow chart of a sequential Monte Carlo loss guidance condition generation module provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention. The parts not described in detail below can be implemented using existing technologies.

[0044] It should be understood that the terms "first", "second", etc. in the following embodiments are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0045] In order to solve the technical problems in the background technology, the controllable generation image processing method proposed in the present invention can be used for image processing in tasks such as text conditional image generation, segmentation map conditional image generation, face map conditional image generation and style map conditional image generation, to generate high-quality images that fully meet the given conditions, and meet the current needs in the field of controllable image generation.

[0046] Reference Figure 1 As shown, an embodiment of the present invention provides an image processing method based on controllable generation, comprising:

[0047] M100, obtaining a conditional signal and an information extraction model of an image processing task, and constructing a first loss function based on the conditional signal and the information extraction model;

[0048] In this step, depending on the specific image processing task, the conditional signal y and the information extraction model H will be different, so as to better meet the task requirements of text conditional image generation, segmentation map conditional image generation, face map conditional image generation and style map conditional image generation.

[0049] Specifically, the information extraction model is a denoising image information extraction model or conditional information extraction model

[0050] In one embodiment, when the image processing task is text conditional image generation: the conditional signal y is a text condition, the information extraction model H is a CLIP model, and the corresponding denoising image information extraction model For CLIP image encoder, conditional information extraction model For the CLIP text encoder.

[0051] In another embodiment, when the image processing task is segmentation map condition image generation: the condition signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model Pre-trained MobileNetV3-Large with segmentation head, conditional information extraction model It is an identical operation.

[0052] In another embodiment, when the image processing task is to generate a face image condition: the condition signal y is a face image condition, the information extraction model H is a face ID feature extraction network, and the corresponding denoised image information extraction model For face ID recognition network, conditional information extraction network For the same Face ID recognition network.

[0053] In another embodiment, when the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoising image information extraction model Get the Gram matrix of features for the CLIP image encoder, conditional information extraction network The Gram matrix of features is obtained for the same CLIP image encoder.

[0054] Correspondingly, the loss functions constructed for different image processing tasks will also be different, and will be determined based on the actual image processing tasks.

[0055] Specifically, in one embodiment, when the image processing task is text conditional image generation: the conditional signal y is the text condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the cosine similarity between y and x0.

[0056] In another embodiment, when the image processing task is segmentation map condition image generation: the condition signal y is the segmentation map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the sum of the cross entropy losses of each pixel between the predicted segmentation maps of y and x0.

[0057] In another embodiment, when the image processing task is to generate a face conditional image: the conditional signal y is a face condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0.

[0058] In another embodiment, when the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

[0059] Of course, the above is an explanation of an embodiment of the present invention. In other possible embodiments, it may also be other image processing tasks, and the information extraction model and the corresponding loss function can be performed according to actual needs, and are not limited to the above description.

[0060] M200, select a pre-trained diffusion model;

[0061] In this embodiment, one of the existing diffusion models that have been pre-trained in the natural image dataset is selected as the diffusion model used in the subsequent steps of the method.

[0062] Specifically, for example, the pre-trained diffusion model can be: selecting any natural image dataset, adding Gaussian noise to the natural image as the diffusion model input, and training with a second loss function that successfully predicts the noise to obtain the pre-trained diffusion model.

[0063] In this step, the second loss function used to predict noise refers to a loss function used when pre-training a general diffusion model, which is different from the first loss function in S100.

[0064] M300, iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image.

[0065] This step specifically includes:

[0066] M301, which samples multiple samples from the initial distribution of Sequential Monte Carlo as a particle filter;

[0067] M302, based on the pre-trained diffusion model and the first loss function, establish an optimized estimate of the suggestion distribution and an unconditional transition probability distribution, and obtain a weight equation of a sequential Monte Carlo;

[0068] M303, resampling to obtain new samples based on the weight equation and resampling strategy of the sequential Monte Carlo method;

[0069] Repeat the above steps M301-M303 until the set number of iterations is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

[0070] In the above embodiment of the present invention, the controllably generated image processing method does not require any data collection and any model training. The pre-trained model in the step can be an existing open source, trained, general model. The iterative optimization used in the above method steps does not involve a training process, that is, the method steps are iterated to optimize the objective function without retraining a diffusion model.

[0071] In another embodiment of the present invention, the image processing method based on controllable generation can be performed according to the following specific steps:

[0072] S100, obtaining a conditional signal y and an information extraction model H of an image processing task, and constructing a first loss function based on the conditional signal and the information extraction model;

[0073] S200, selecting a pre-trained diffusion model, wherein the pre-trained diffusion model may be trained in a natural image dataset using a loss function for predicting noise.

[0074] S300, sampling multiple samples from the initial distribution of Sequential Monte Carlo as a particle filter;

[0075] For example, define T as the total number of iterations, set the mean to 0, and the variance to T 2 Gaussian distribution N(0,T 2 ) is the initial distribution p(x) of the Sequential Monte Carlo T ), sample N initialization samples from the distribution as particle filters.

[0076] In this step, the particle filter is a recursive filter using the Monte Carlo method, which uses a set of weighted random samples (called particles) to represent the posterior probability of random events and estimates the state of the dynamic system from a noisy or incomplete observation sequence.

[0077] S400, based on the pre-trained diffusion model and the first loss function, establish an optimization estimate of the proposal distribution and an unconditional transition probability distribution, and obtain a weight equation of the sequential Monte Carlo;

[0078] In this step, in order to utilize loss-guided diffusion sampling to achieve asymptotically accurate generation under various conditions such as text, style, face images, segmentation images, etc., and to generate high-quality images that fully meet the given conditions, an optimized estimate of the proposal distribution and an unconditional transition probability distribution are established based on the diffusion model and loss function.

[0079] Specifically, in some preferred embodiments, this step can establish an optimized estimate of the proposed distribution and an unconditional transition probability distribution according to the following operations:

[0080] Let the nth sample x in the tth iteration be t , define the diffusion model as x θ (.), θ represents the model parameters;

[0081] S401, using diffusion model x θ (.) From input x t Get the unconditional score function s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t is the standard deviation of the noise in the t-th iteration;

[0082] S402, calculating the conditional score function s θ (x t ,y)=s θ (x t )-αL(y,x θ (xt )), where α is the optimization step size and L is the constructed loss function;

[0083] S403, calculate the likelihood probability distribution p(y|x θ (x t ))=exp(-L(y,x θ (x t ))) / Z, where exp represents the exponential function and Z is a normalization constant greater than 0;

[0084] S404, based on the conditional score function s θ (x t ,y), build an optimal estimate of the proposed distribution And the unconditional transition probability distribution Where N represents Gaussian distribution and I represents the identity matrix.

[0085] S405, based on the optimization estimation of the proposed distribution r(x t-1 |x t ) and the unconditional transition probability distribution p(x t-1 |x t ), and obtain the weight equation of Sequential Monte Carlo

[0086] This embodiment adopts the above steps S401-S405, and can flexibly construct different loss functions according to different controllable generation tasks, so that all tasks can be unified into one framework, which can overcome the problem that the prior art requires separate construction for each task and the cumbersome construction process.

[0087] S500, based on the weight equation and resampling strategy of Sequential Monte Carlo, resampling is performed to obtain new samples;

[0088] In this step, the resampling strategy of Sequential Monte Carlo is used to resample the total N samples and the corresponding weight equation to obtain new N samples. The specific resampling strategy can refer to the existing technology of Sequential Monte Carlo.

[0089] For example, the N samples initialized in S300 can be regarded as the input at the Tth iteration. After the N initialized samples are input at the Tth time, after the steps of S400 and S500, N processed samples at the T-1th time are obtained, which are the obtained new samples.

[0090] S600, repeating the above S300 to S500 until the set number of iteration steps is reached, and a final weight equation is obtained. The last sampled multiple samples corresponding to the final weight equation are the final generated image.

[0091] Following the example of S500 above, it can be considered that each sample has two indexes, namely time t (from 1 to T) and sequence number n (from 1 to N), and the entire iteration process is iterated in sequence from T to 1. During the iteration process, samples with different sequence numbers (from 1 to N) are processed independently and in parallel in each round of iteration.

[0092] The initialized N samples will continuously iterate based on steps S400 and S500, and generate N new samples corresponding to time T-1, ..., 2, 1 in turn. When the set number of iterations is reached, time 1 will be reached, and the final weight equation is the weight equation of t=1 in S405. The final generated image is the effect of the N samples corresponding to time 1, and the best effect is selected as the output result.

[0093] In the above-mentioned embodiment of the present invention, based on the diffusion model and sequential Monte Carlo (SMC), asymptotically accurate loss-guided diffusion sampling is implemented, and the pre-trained encoder of the information extraction model is used to model the likelihood function according to the similarity of the feature space and the energy-based function, and universal controllable generation is achieved under a wide range of conditions. It is possible to achieve diverse high-quality image conditional generation without training, meeting the needs of general processing.

[0094] In a preferred implementation of the present invention, taking a method for generating conditional images of human faces based on controllable generation as an example, the specific steps are the same as those of the image processing method based on controllable generation in the above-mentioned embodiment, and the differences in the implementation of the task of generating conditional images of human faces are described in detail as follows.

[0095] like Figure 2 As shown, the image processing task performed in this specific embodiment is to generate a face conditional image, which specifically includes the following steps:

[0096] S101, obtain input, obtain N samples in total, determine the input value of N samples according to the number of iteration steps, if the iteration time is T, that is, the first iteration time, from the normal distribution P(x T )=N(0,T 2 I) is used as input for sampling noise; if the iteration time is not T, then the series of samples obtained in the previous step Neutralization weight equation For input.

[0097] S102, a series of samples obtained in the previous step According to the weight equation Resample to get new N samples

[0098] S103, using the diffusion model to solve. Specifically, receiving the output of the diffusion model, that is, the noisy sample x at time t t , through the pre-trained diffusion model x θ (.), output without noise sample x θ (x t ).

[0099] S104, the sequential Monte Carlo loss guided condition generation module receives the output x of the diffusion model θ (x t ), using the sequential Monte Carlo algorithm combined with the loss function to guide the sampling process, we get the proposed distribution r(x t-1 |x t ) and the weight equation for Sequential Monte Carlo.

[0100] The conditions and loss functions used in S104 above correspond to the face image generation task. In this step, the sequential Monte Carlo loss-guided condition generation module is equivalent to the processing content in step S400 of the previous embodiment, that is, the sequential Monte Carlo algorithm is combined with the loss function to guide the sampling process to obtain the proposed distribution r(x t-1 |x t ) and the weight equation of Sequential Monte Carlo. In simple terms, it is to obtain the optimized estimate (or optimized sample) and weight equation of the proposed distribution for each iterative step.

[0101] For example, multiple samples are sampled from the initial distribution of the sequential Monte Carlo, where N samples are initialized as particle filters and as processing samples for the subsequent sequential Monte Carlo loss guidance condition generation module. The N samples obtained by initialization can be regarded as the input at the Tth iteration. The iteration of the diffusion method is in reverse order, that is, starting from T and ending at 1. After these N initialized samples are input at the Tth moment, after step S400, N processed samples are obtained at the T-1th moment. At each time t (t is any integer between 1 and T), for each sample n (n is any integer between 1 and N), the steps executed are consistent with the description of step S400. The processing of these N samples is independent.

[0102] As an embodiment, according to Figure 3 The detailed flow chart of the sequential Monte Carlo loss guided condition generation module is shown, S104 may include the following steps:

[0103] S1041, receiving the output of the pre-trained diffusion model, i.e., the noise-free sample x θ (x t ), and the conditional signal y, as input for subsequent processing of the module.

[0104] S1042, the output of the pre-trained diffusion model, i.e., the noise-free sample x θ (x t ), processed by the conditional information extraction model H, the features of the denoised output are obtained And the characteristics of the conditional signal In the case of face-conditional image generation, For face ID recognition network, For the same Face ID recognition network.

[0105] S1043, calculate the loss function of a single step,

[0106] S1044, calculate the optimal estimate of the proposed distribution, based on the output x of the diffusion model θ (x t ), and obtain the unconditional score equation s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t is the standard deviation of the noise at time step t;

[0107] According to Tweedie theorem combined with the loss function L(y,x θ (x t )), we get the conditional score function, s θ (x t ,y)=s θ (x t )-αL(y,x θ (x t )), where α is the optimization step size.

[0108] According to the loss function L(y,x θ (x t )) Combined with the definition of the energy equation, we get the likelihood probability distribution p(y|x θ (x t ))=exp(-L(y,x θ (x t ))) / Z. According to the conditional score function, we get the proposed distribution r(x t-1 |x t ), that is, At the same time, we get the unconditional transition probability distribution

[0109] S1045, calculating the weight equation of the sequential Monte Carlo. Specifically, the weight equation of the sequential Monte Carlo is obtained by calculating the optimal estimate of the proposed distribution, the unconditional transition probability distribution, and the likelihood probability distribution, that is,

[0110] This step S104 is a refined sequential Monte Carlo-based loss function guided diffusion model condition generation of the present invention. Through this strategy, the quality of the generated image and the compliance with the given conditions are improved.

[0111] S105, determine whether the current iteration time t is equal to the last iteration time. If t is equal to 1, the optimized result obtained in step S104 is used as the final output sample series, and the final optimal sample is obtained according to the weight equation and the sample series. If t is not equal to 1, the optimized sample series and the weight equation obtained in step S105 are used as the input of the next iteration time t.

[0112] The weight equation is an important algorithm equation of Sequential Monte Carlo. The recorded intermediate quantities are used for resampling. The optimized result is the optimized image sample obtained in each iteration step. The obtained sample series means that there are multiple samples, corresponding to the initialization of N samples. The final optimal sample is the one with the best effect among the N samples corresponding to the evaluation time 1, and that one is selected as the output result, that is, only the best sample among the N samples obtained by the final optimization is selected as the final output result.

[0113] Based on the same technical concept as above, in another embodiment of the present invention, an image processing system of a controllably generated image processing method is provided, which is used to implement the image processing method in the above embodiment, and to achieve tasks such as segmentation map conditional image generation, face map conditional image generation, and style map conditional image generation. Specifically, the image processing system includes:

[0114] Data acquisition module: obtains the conditional signal y and information extraction model H of the image processing task;

[0115] A construction module: constructing a first loss function based on the conditional signal and information extraction model obtained by the data acquisition module, and at the same time, constructing a diffusion model, and training using a second loss function for predicting noise in an image data set to obtain a pre-trained diffusion model;

[0116] Image processing module: iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image;

[0117] Wherein, the image processing module includes:

[0118] Module M1, which samples multiple samples from the initial distribution of Sequential Monte Carlo, acts as a particle filter;

[0119] Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the suggestion distribution and an unconditional transition probability distribution to obtain a weight equation of a sequential Monte Carlo;

[0120] Module M3, based on the weight equation and resampling strategy of the sequential Monte Carlo, resampling to obtain new samples;

[0121] The above modules M1-M3 are executed in sequence until the set number of iterations is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

[0122] The module M2 included in the above-mentioned image processing module corresponds to the sequential Monte Carlo loss guided condition generation module in the above-mentioned method embodiment. The module M2 (sequential Monte Carlo loss guided condition generation module) is included in the image processing module and is a key step in each iteration. It obtains the optimized sample and weight equation of one step in the iteration; the module M2 (sequential Monte Carlo loss guided condition generation module) obtains data from the data acquisition module and obtains the pre-trained model to be used in each step from the model construction model.

[0123] The specific implementation technology of each module of the controllably generated image processing system in the above embodiments of the present invention can refer to the specific description of the steps in any of the above controllably generated image processing method embodiments, which will not be repeated here.

[0124] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to run the above-mentioned system when executing the program.

[0125] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.

[0126] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0127] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0128] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method, or run the above system.

[0129] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image processing method based on controllable generation, characterized in that: include: Acquire a conditional signal and an information extraction model of an image processing task, and construct a first loss function based on the conditional signal and the information extraction model; Select a pre-trained diffusion model; Iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image; specifically comprising: M1, samples multiple samples from the initial distribution of Sequential Monte Carlo as a particle filter; M2, based on the pre-trained diffusion model and the first loss function, establish an optimized estimate of the suggestion distribution and an unconditional transition probability distribution to obtain a weight equation of a sequential Monte Carlo; M3, resampling to obtain new samples based on the weight equation and resampling strategy of the sequence Monte Carlo; Repeat the above steps M1-M3 until the set number of iterations is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

2. The image processing method based on controllable generation according to claim 1, characterized in that: The information extraction model is a denoising image information extraction model or conditional information extraction model The conditional signal and information extraction model for obtaining the image processing task are specifically any one of the following: -When the image processing task is text conditional image generation: the conditional signal y is a text condition, the information extraction model H is a CLIP model, and the corresponding denoising image information extraction model For CLIP image encoder, conditional information extraction model It is the CLIP text encoder; -When the image processing task is segmentation map condition image generation: the conditional signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model Pre-trained MobileNetV3-Large with segmentation head, conditional information extraction model is an identity operation; -When the image processing task is to generate a face image condition: the conditional signal y is the face image condition, the information extraction model H is the face ID feature extraction network, and the corresponding denoising image information extraction model For face ID recognition network, conditional information extraction network For the same face ID recognition network; -When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoising image information extraction model Get the Gram matrix of features for the CLIP image encoder, conditional information extraction network The Gram matrix of features is obtained for the same CLIP image encoder.

3. The image processing method based on controllable generation according to claim 1, characterized in that: A first loss function is constructed based on the conditional signal and the information extraction model, specifically any one of the following: -When the image processing task is text conditional image generation: the conditional signal y is the text condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the cosine similarity between y and x0; -When the image processing task is segmentation map condition image generation: the conditional signal y is the segmentation map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the sum of the cross entropy losses of each pixel between the predicted segmentation maps of y and x0; -When the image processing task is to generate a face image condition: the conditional signal y is a face image condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0; -When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

4. The image processing method based on controllable generation according to claim 1, characterized in that: The method of sampling a plurality of samples from the initial distribution of the sequence Monte Carlo as a particle filter comprises: Define T as the total number of iterations, set the mean to 0 and the variance to T 2 The Gaussian distribution of is the initial distribution of the sequence Monte Carlo, and N initialization samples are sampled from the distribution as the particle filter.

5. The image processing method based on controllable generation according to claim 1, characterized in that: Based on the pre-trained diffusion model and the first loss function, an optimized estimate of a suggestion distribution and an unconditional transition probability distribution are established, including: Let the nth sample x in the tth iteration be t , define the diffusion model as x θ (.), θ represents the model parameters; Using the diffusion model x θ (.) From the input x t Get the unconditional score function s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t is the standard deviation of the noise in the t-th iteration; Calculate the conditional score function s θ (x t ,y)=s θ (x t )-αL(y,x θ (x t )), where α is the optimization step size and L is the constructed loss function; Calculate the likelihood probability distribution p(y|x θ (x t ))=exp(-L(y,x θ (x t ))) / Z, where exp represents the exponential function and Z is a normalization constant greater than 0; Based on the conditional score function s θ (x t ,y), build an optimal estimate of the proposed distribution And the unconditional transition probability distribution Where N represents Gaussian distribution and I represents the identity matrix.

6. The image processing method based on controllable generation according to claim 5, characterized in that: The method of obtaining a weight equation of Sequential Monte Carlo includes: Optimization based on the proposed distribution estimates r(x t-1 |x t ) and the unconditional transition probability distribution p(x t-1 |x t ), and obtain the weight equation of Sequential Monte Carlo 7. The image processing method based on controllable generation according to claim 1, characterized in that: Based on the weight equation and resampling strategy of the Sequential Monte Carlo, resampling is performed to obtain new samples, including: Using the Sequential Monte Carlo resampling strategy, resampling is performed based on a total of N samples and the corresponding weight equation to obtain new N samples.

8. An image processing system based on controllable generation, characterized in that: include: Data acquisition module: obtains the conditional signal y and information extraction model H of the image processing task; A construction module: constructing a first loss function based on the conditional signal and information extraction model obtained by the data acquisition module, and at the same time, constructing a diffusion model, and training using a second loss function for predicting noise in an image data set to obtain a pre-trained diffusion model; Image processing module: iterating based on the pre-trained diffusion model and the first loss function to obtain a final generated image; Wherein, the image processing module includes: Module M1, which samples multiple samples from the initial distribution of Sequential Monte Carlo, acts as a particle filter; Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the suggestion distribution and an unconditional transition probability distribution to obtain a weight equation of a sequential Monte Carlo; Module M3, based on the weight equation and resampling strategy of the sequential Monte Carlo, resampling to obtain new samples; The above modules M1-M3 are executed in sequence until the set number of iterations is reached, and the final weight equation and the final sampled multiple samples are obtained to obtain the final generated image.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images