Image processing method, system, terminal and medium based on controllable generation

Abstract

This invention provides a controllable image processing method, system, terminal, and medium, comprising: acquiring conditional signals and information extraction models for the image processing task, and constructing a first loss function; selecting a pre-trained diffusion model; and iterating based on the diffusion model and the first loss function, including: M1, sampling multiple samples from the initial distribution of the sequence Monte Carlo; M2, establishing an optimized estimate of the proposed distribution and an unconditional transition probability distribution to obtain the weight equation of the sequence Monte Carlo; M3, resampling to obtain new samples; and repeating M1-M3 until a set number of iterations is reached to obtain the final weight equation and the sampled multiple samples, resulting in the final generated image. This invention can achieve high-quality, diverse, and training-free conditional image generation in tasks such as text conditional image generation, segmentation map conditional image generation, face map conditional image generation, and style map conditional image generation, meeting the needs of general processing.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to an image processing method and system, terminal, and medium based on controllable generation. Background Technology

[0002] With the continuous advancement of media technology, diffusion models have demonstrated superiority over existing generative models in generative tasks, such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Normalizing Flows. The powerful generative capabilities of diffusion models have driven their widespread application in numerous downstream tasks. Among these applications, conditional generation, due to its ability to generate images under controlled conditions, has become a highly promising area. Many conditional diffusion models have been optimized for various conditions, including category labels, text prompts, degraded images, reference images, segmentation maps, and style images.

[0003] Methods for conditional generation using diffusion models are generally categorized into training-based and training-free methods. Training-based strategies require retraining the model based on given conditions. While these methods can deliver excellent generative performance, they lack flexibility, especially when faced with new conditions; retraining or fine-tuning the model is costly and results in poor adaptability. In contrast, training-free conditional generation methods aim to overcome the challenges of additional training. These methods utilize readily available loss functions combined with pre-trained (unconditional or conditional) diffusion models to guide the sampling process. For example, methods such as "FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model" presented at the 2023 Conference on Computer Vision and Pattern Recognition (CVPR), and "Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation" and "Universal Guidance for Diffusion Models" presented at the 2023 International Conference on Machine Learning (ICML) all address the problem of conditionally uncertain image generation by guiding conditional sampling through implicit priors and conditional loss functions of the diffusion model, without requiring model retraining. However, these methods still suffer from shortcomings in consistency and reliability; the generated images may not fully conform to the given conditions. This is because their heuristic approximations of the conditional fractional function (such as the Laplace approximation) may introduce errors. Summary of the Invention

[0004] This invention addresses the problems of existing diffusion-based, training-free conditional generation techniques, which fail to generate images that fully meet given conditions and produce low-quality images. It provides an image processing method, system, terminal, and medium based on controllable generation, capable of generating high-quality images that fully meet given conditions, thus satisfying the current needs in the field of controllable image generation.

[0005] According to a first aspect of the present invention, a controllable image processing method is provided, comprising:

[0006] Obtain the conditional signals and information extraction model for the image processing task, and construct a first loss function based on the conditional signals and the information extraction model;

[0007] Choose a pre-trained diffusion model;

[0008] The final generated image is obtained by iterating based on the pre-trained diffusion model and the first loss function;

[0009] Wherein: based on the pre-trained diffusion model and the first loss function, the final generated image is obtained through iteration, including:

[0010] M1 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter;

[0011] M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the proposed distribution and an unconditional transition probability distribution, and obtains the sequence Monte Carlo weight equation;

[0012] M3, based on the Monte Carlo weighting equation and resampling strategy of the sequence, performs resampling to obtain new samples;

[0013] Repeat steps M1-M3 until the set number of iterations is reached to obtain the final weight equation and the final sampled multiple samples, resulting in the final generated image.

[0014] This invention enables the generation of diverse, high-quality images without training in tasks such as text-based conditional image generation, segmentation-based conditional image generation, face-based conditional image generation, and style-based conditional image generation, thus meeting the needs of general processing.

[0015] Preferably, the conditional signal y for the image processing task and the information extraction model H are obtained, determined according to the task type. The information extraction model H includes: a denoised image information extraction model. Conditional information extraction model include:

[0016] - When the image processing task is text-conditional image generation: the conditional signal y is the text condition, the information extraction model H is a CLIP model, and the corresponding denoised image information extraction model... Conditional information extraction model for CLIP image encoder For CLIP text encoder;

[0017] - When the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model is... For pre-trained MobileNetV3-Large with segmentation head, conditional information extraction model This is an identity operation;

[0018] - When the image processing task is face image conditional image generation: the conditional signal y is the face image condition, the information extraction model H is the face ID feature extraction network, and the corresponding denoising image information extraction model is... For face ID recognition networks, conditional information extraction networks For the same face ID recognition network;

[0019] - When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoised image information extraction model... To obtain the Gram matrix of features for the CLIP image encoder, a conditional information extraction network is used. The same CLIP image encoder obtains the Gram matrix of features.

[0020] Preferably, the loss function is constructed based on the conditional signal and information extraction network, including:

[0021] - When the image processing task is text-conditional image generation: the conditional signal y is the text condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the cosine similarity between y and x0;

[0022] - When the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, x0 is defined as the final generated image, and the loss function L(y,x0) is the sum of the cross-entropy loss of each pixel between the predicted segmentation maps of y and x0;

[0023] - When the image processing task is face image conditional image generation: the conditional signal y is the face image condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0;

[0024] - When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the final generated image, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

[0025] According to a second aspect of the present invention, a controllable image processing system is characterized by comprising:

[0026] Data acquisition module: Acquires the conditional signal y and information extraction model H for the image processing task;

[0027] Construction module: Based on the conditional signal and information extraction model obtained by the data acquisition module, a first loss function is constructed. At the same time, a diffusion model is constructed and trained on the image dataset using a second loss function for predicting noise to obtain a pre-trained diffusion model.

[0028] Image processing module: Iterates based on the pre-trained diffusion model and the first loss function to obtain the final generated image;

[0029] The image processing module includes:

[0030] Module M1 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter;

[0031] Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the proposed distribution and an unconditional transition probability distribution, and obtains the sequence Monte Carlo weight equation.

[0032] Module M3, based on the weighting equation and resampling strategy of the sequence Monte Carlo, performs resampling to obtain new samples;

[0033] The modules M1-M3 are executed sequentially until the set number of iterations is reached, resulting in the final weight equation and multiple samples, thus producing the final generated image.

[0034] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described herein, or to run the system described herein.

[0035] According to a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described herein, or to run the system described herein.

[0036] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0037] The image processing method and system based on controllable generation provided in this invention are based on a training-free Sequence Monte Carlo (SMC) method, achieving controllable generation through a diffusion model. Compared to existing training-free diffusion model methods, this method utilizes loss-guided diffusion sampling to achieve asymptotically accurate generation under various conditions such as text, style, face image, and segmentation map, generating high-quality images that fully meet the given conditions, thus satisfying the current needs in the field of controllable image generation.

[0038] The image processing method and system based on controllable generation provided in this invention have a plug-and-play flexibility compared to deep neural network methods based on supervised learning. They do not require additional training, data collection, or preprocessing, thus saving computational and human resources. Attached Figure Description

[0039] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0040] Figure 1 This is a flowchart of the overall process of an image processing method based on controllable generation in one embodiment of the present invention;

[0041] Figure 2 The overall flowchart for the controllable generation of a loss-guided diffusion model provided in a preferred embodiment of the present invention is shown.

[0042] Figure 3 This is a detailed flowchart of a sequence Monte Carlo loss guidance condition generation module provided in a preferred embodiment of the present invention. Detailed Implementation

[0043] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention. Parts not described in detail below can be implemented using existing technology.

[0044] It should be understood that the terms "first," "second," etc., in the following embodiments are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0045] To address the technical problems in the background art, the image processing method for controllable generation proposed in this invention can be used for image processing tasks such as text-conditional image generation, segmentation map-conditional image generation, face map-conditional image generation, and style map-conditional image generation, generating high-quality images that fully meet the given conditions, thus satisfying the current needs in the field of controllable image generation.

[0046] Reference Figure 1 As shown, this embodiment of the invention provides an image processing method based on controllable generation, comprising:

[0047] M100 acquires the conditional signals and information extraction model for the image processing task, and constructs a first loss function based on the conditional signals and the information extraction model;

[0048] In this step, the conditional signal y and the information extraction model H will vary depending on the specific image processing task, so as to better meet the requirements of tasks such as text conditional image generation, segmentation map conditional image generation, face map conditional image generation, and style map conditional image generation.

[0049] Specifically, the information extraction model is a denoised image information extraction model. Or conditional information extraction model

[0050] In one embodiment, when the image processing task is text-conditional image generation: the conditional signal y is the text condition, the information extraction model H is a CLIP model, and the corresponding denoised image information extraction model... Conditional information extraction model for CLIP image encoder This is the CLIP text encoder.

[0051] In another embodiment, when the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model... For pre-trained MobileNetV3-Large with segmentation head, conditional information extraction model This is an identity operation.

[0052] In another embodiment, when the image processing task is face image conditional image generation: the conditional signal y is the face image condition, the information extraction model H is the face ID feature extraction network, and the corresponding denoising image information extraction model... For face ID recognition networks, conditional information extraction networks For the same face ID recognition network.

[0053] In another embodiment, when the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoised image information extraction model... To obtain the Gram matrix of features for the CLIP image encoder, a conditional information extraction network is used. The same CLIP image encoder obtains the Gram matrix of features.

[0054] Correspondingly, the loss function will vary depending on the specific image processing task, and should be determined based on the actual image processing task.

[0055] Specifically, in one embodiment, when the image processing task is text-conditional image generation: the conditional signal y is the text condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the cosine similarity between y and x0.

[0056] In another embodiment, when the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the sum of the cross-entropy loss of each pixel between the predicted segmentation maps of y and x0.

[0057] In another embodiment, when the image processing task is face image conditional image generation: the conditional signal y is the face image condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0.

[0058] In another embodiment, when the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

[0059] Of course, the above is a description of the embodiments of the present invention. In other possible embodiments, other image processing tasks may also be used. Information extraction models and corresponding loss functions can be implemented according to actual needs, and are not limited to the above description.

[0060] M200 selects a pre-trained diffusion model;

[0061] In this embodiment, one of the existing diffusion models that have been pre-trained in a natural image dataset is selected as the diffusion model to be used in the subsequent steps of this method.

[0062] Specifically, for example, a pre-trained diffusion model can be obtained by selecting any natural image dataset, adding Gaussian noise to the natural images as input to the diffusion model, and training it with a second loss function that successfully predicts the noise.

[0063] In this step, the second loss function used to predict noise refers to the loss function used when pre-training a general diffusion model, which is different from the first loss function in S100.

[0064] M300 iterates based on the pre-trained diffusion model and the first loss function to obtain the final generated image.

[0065] This step specifically includes:

[0066] M301 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter;

[0067] M302, based on the pre-trained diffusion model and the first loss function, establish the optimized estimate of the proposed distribution and the unconditional transition probability distribution, and obtain the sequence Monte Carlo weight equation;

[0068] M303, based on the Monte Carlo weighting equation and resampling strategy of the sequence, resampling is performed to obtain new samples;

[0069] Repeat steps M301-M303 until the set number of iterations is reached to obtain the final weight equation and the final sampled multiple samples, resulting in the final generated image.

[0070] In the above embodiments of the present invention, the controllable image processing method does not require any data collection or model training. The pre-trained model in the steps can be an existing open-source, pre-trained, and general model. The iterative optimization in the above method steps does not involve the training process. That is, the method steps are iterated to optimize the objective function without retraining a diffusion model.

[0071] In another embodiment of the present invention, the image processing method based on controllable generation can be performed according to the following specific steps:

[0072] S100: Obtain the conditional signal y and information extraction model H for the image processing task, and construct the first loss function based on the conditional signal and information extraction model;

[0073] S200, Select a pre-trained diffusion model, wherein the pre-trained diffusion model can be trained on a natural image dataset using a loss function that predicts noise.

[0074] S300 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter;

[0075] For example, let T be the set total number of iterations, with a mean of 0 and a variance of T. 2 Gaussian distribution N(0,T) 2 p(x) is the initial distribution of the Monte Carlo sequence. T N initial samples are obtained from this distribution and used as particle filters.

[0076] In this step, the particle filter is a recursive filter that uses the Monte Carlo method to represent the posterior probability of random events with a set of weighted random samples (called particles) to estimate the state of a dynamic system from a noisy or incomplete sequence of observations.

[0077] S400, based on the pre-trained diffusion model and the first loss function, establishes the optimal estimate of the proposal distribution and the unconditional transition probability distribution, and obtains the weight equation of sequence Monte Carlo.

[0078] In this step, in order to utilize loss-guided diffusion sampling to achieve asymptotically accurate generation under various conditions such as text, style, face image, and segmentation image, and to generate high-quality images that fully meet the given conditions, an optimized estimate of the proposal distribution and an unconditional transition probability distribution are established based on the diffusion model and loss function.

[0079] Specifically, in some preferred embodiments, this step can be performed as follows to establish an optimized estimate of the proposed distribution and an unconditional transition probability distribution:

[0080] Let x be the nth sample in the t-th iteration. t The diffusion model is defined as x θ (.), where θ represents the model parameters;

[0081] S401, using diffusion model x θ (.) From input x t Obtain the unconditional score function s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t Let be the standard deviation of the noise in the t-th iteration;

[0082] S402, Calculate the conditional score function s θ (x t ,y)=s θ (x t )-αL(y,x θ (xt ), where α is the optimization step size and L is the constructed loss function;

[0083] S403, Calculate the likelihood probability distribution p(y|x) θ (x t ))=exp(-L(y,x θ (x t ))) / Z, where exp represents the exponential function and Z is a normalization constant greater than 0;

[0084] S404, based on conditional scoring function s θ (x t ,y), establish an optimal estimate of the proposed distribution. and the unconditional transition probability distribution Where N represents a Gaussian distribution and I represents the identity matrix.

[0085] S405, Optimal estimation of r(x) based on the proposed distribution t-1 |x t and the unconditional transition probability distribution p(x) t-1 |x t The weight equations for sequence Monte Carlo are obtained.

[0086] This embodiment uses steps S401-S405 as described above, which can flexibly construct different loss functions according to different controllable generation tasks, thereby unifying all tasks into a framework. This can overcome the problem that existing technologies require separate construction for each task and the construction process is cumbersome.

[0087] S500, based on the weighting equation and resampling strategy of sequence Monte Carlo, performs resampling to obtain new samples;

[0088] In this step, a sequence Monte Carlo resampling strategy is used to resample the N samples based on the corresponding weight equations, resulting in N new samples. The specific resampling strategy can be found in existing sequence Monte Carlo techniques.

[0089] For example, the N samples obtained from the initialization in S300 can be regarded as the input at the T-th iteration. After these N initial samples are input at time T, they are processed through steps S400 and S500 to obtain the N processed samples at time T-1, which are the new samples.

[0090] S600, repeat S300 to S500 until the set number of iterations is reached, and the final weight equation is obtained. The last sampled multiple samples corresponding to the final weight equation are the final generated image.

[0091] Continuing with the S500 example above, we can assume that each sample has two indices: time t (from 1 to T) and sequence number n (from 1 to N). The entire iteration process proceeds sequentially from time T to 1. During the iteration, samples with different sequences (from 1 to N) are processed independently and in parallel in each iteration.

[0092] The initial N samples are iterated continuously based on steps S400 and S500, generating new N samples corresponding to times T-1, ..., T2, T1. When the set number of iterations is reached, time 1 is achieved, and the final weight equation is the weight equation for t=1 in S405. The final generated image evaluates the performance of the N samples corresponding to time 1, selecting the best-performing image as the output.

[0093] In the above embodiments of the present invention, based on the diffusion model and Sequential Monte Carlo (SMC), diffusion sampling guided by asymptotically accurate loss is achieved. The pre-trained encoder of the information extraction model is used to model the likelihood function according to the similarity of the feature space and the energy-based function, and general controllable generation is achieved under a wide range of conditions. It can achieve diverse high-quality image condition generation without training and meet the needs of general processing.

[0094] In a preferred embodiment of the present invention, taking the face image conditional image generation method based on controllable generation as an example, the specific steps are the same as the image processing method based on controllable generation in the above embodiments. The differences in the implementation of the face image conditional image generation task are described in detail below.

[0095] like Figure 2 As shown, the image processing task performed in this specific embodiment is face image conditional image generation, which specifically includes the following steps:

[0096] S101, Obtain the input. A total of N samples are obtained. The input values ​​of the N samples are determined according to the number of iterations. If the iteration time is T, which is the first iteration time, the input values ​​are obtained from the normal distribution P(x). T )=N(0,T 2 In step I), noise is sampled and used as input; if the iteration time is not T, then the series of samples obtained in the previous step... Neutralization weight equation For input.

[0097] S102 then takes the series of samples obtained in the previous step According to the weight equation Resampling is performed to obtain N new samples.

[0098] S103 is solved using a diffusion model. Specifically, it receives the output of the diffusion model, i.e., the noisy sample x at time t. t Through a pre-trained diffusion model x θ (.), outputs a noise-free sample x. θ (x t ).

[0099] S104, the Sequence Monte Carlo Loss Guided Condition Generation Module receives the output x of the diffusion model. θ (x t Using the sequential Monte Carlo algorithm combined with a loss function to guide the sampling process, the proposed distribution r(x) is obtained. t-1 |x t ) optimization estimation and sequence Monte Carlo weight equation.

[0100] The conditions and loss function used in S104 above correspond to the face image generation task. In this step, the sequential Monte Carlo loss-guided condition generation module is equivalent to the processing content in step S400 of the previous embodiment, that is, using the sequential Monte Carlo algorithm combined with the loss function to guide the sampling process to obtain the proposed distribution r(x). t-1 |x t The optimization estimate of the distribution and the weight equation for the sequence Monte Carlo method. Simply put, it involves obtaining an optimization estimate (or optimized sample) of the proposed distribution and the weight equation for each iteration step.

[0101] For example, multiple samples are sampled from the initial distribution of the sequence Monte Carlo algorithm. N samples are initialized as particle filters, serving as processing samples for the subsequent sequence Monte Carlo loss-guided condition generation module. These N initialized samples can be considered as input at iteration step T. The diffusion method iterates in reverse order, starting from T and ending at 1. After these N initialized samples are input at time T, they undergo step S400 to obtain N processed samples at time T-1. For each sample n (n is any integer between 1 and T) at each time t (t takes any value between 1 and T), the steps performed are consistent with those described in step S400. The processing of these N samples is independent.

[0102] As an example, according to Figure 3 The detailed flowchart of the sequence Monte Carlo loss guided condition generation module is shown. S104 may include the following steps:

[0103] S1041, Receive the output of the pre-trained diffusion model, i.e., the noise-free sample x. θ (x t ), and the condition signal y, serve as inputs for subsequent processing by the module.

[0104] S1042, the output of the pre-trained diffusion model, i.e., the noise-free sample x, is... θ (x t The denoised output features are obtained by processing the data through the conditional information extraction model H. and the characteristics of conditional signals In the case of conditional image generation of face images For facial ID recognition networks, For the same face ID recognition network.

[0105] S1043, Calculate the loss function for a single step.

[0106] S1044, Calculate the optimal estimate of the proposed distribution based on the output x of the diffusion model. θ (x t ), thus obtaining the unconditional scoring equation s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t Let be the noise standard deviation at time step t;

[0107] Based on Tweedie's theorem and the loss function L(y,x) θ (x t This yields a conditional score function, s. θ (x t ,y)=s θ (x t )-αL(y,x θ (x t ), where α is the optimization step size.

[0108] Based on the loss function L(y,x) θ (x t Combining this with the definition of the energy equation, we obtain the likelihood probability distribution p(y|x). θ (x t ))=exp(-L(y,x θ (x t Based on the conditional scoring function, the proposed distribution r(x) is obtained. t-1 |x t The optimal estimate of ) is, i.e. At the same time, the unconditional transition probability distribution is obtained.

[0109] S1045, calculate the Monte Carlo weight equation for the sequence. Specifically, the Monte Carlo weight equation for the sequence is obtained through the optimal estimation of the proposal distribution, the unconditional transition probability distribution, and the likelihood probability distribution.

[0110] Step S104 is the refined sequential Monte Carlo-based loss function-guided diffusion model conditional generation of the present invention. This strategy improves the quality of the generated image and its compliance with given conditions.

[0111] S105, determine whether the current iteration time t is equal to the last iteration time. If t equals 1, use the optimized result obtained in step S104 as the final output sample series, and obtain the final optimal sample based on the weight equation and the sample series. If t is not equal to 1, use the optimized sample series obtained in step S105 and the weight equation as the input for the next iteration time t.

[0112] The weighting equation is a crucial algorithmic equation in sequential Monte Carlo simulation. Recorded intermediate values ​​are used for resampling. The optimized result is the optimized image sample obtained at each iteration step. The resulting sample series refers to having multiple samples, corresponding to N initial samples. The final optimal sample is selected as the output from the N samples corresponding to evaluation time 1, meaning only the best sample from the final optimized N samples is chosen as the final output.

[0113] Based on the same technical concept described above, in another embodiment of the present invention, an image processing system is provided for a controllable image processing method. This system implements the image processing method described above, achieving tasks such as segmentation map conditional image generation, face map conditional image generation, and style map conditional image generation. Specifically, the image processing system includes:

[0114] Data acquisition module: Acquires the conditional signal y and information extraction model H for the image processing task;

[0115] Construction module: Based on the conditional signal and information extraction model obtained by the data acquisition module, a first loss function is constructed. At the same time, a diffusion model is constructed and trained on the image dataset using a second loss function for predicting noise to obtain a pre-trained diffusion model.

[0116] Image processing module: Iterates based on the pre-trained diffusion model and the first loss function to obtain the final generated image;

[0117] The image processing module includes:

[0118] Module M1 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter;

[0119] Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the proposed distribution and an unconditional transition probability distribution, and obtains the sequence Monte Carlo weight equation.

[0120] Module M3, based on the weighting equation and resampling strategy of the sequence Monte Carlo, performs resampling to obtain new samples;

[0121] The modules M1-M3 are executed sequentially until the set number of iterations is reached, resulting in the final weight equation and multiple samples, thus generating the final image.

[0122] The image processing module mentioned above includes module M2, which corresponds to the sequence Monte Carlo loss guidance condition generation module in the above method embodiment. Module M2 (sequence Monte Carlo loss guidance condition generation module) is included in the image processing module and is a key step in each iteration, obtaining the optimized samples and weight equations for each iteration. Module M2 (sequence Monte Carlo loss guidance condition generation module) obtains data from the data acquisition module and obtains the pre-trained model to be used in each step from the model construction module.

[0123] The specific implementation techniques of each module of the controllable image processing system in the above embodiments of the present invention can be referred to the detailed description of the steps in any of the above embodiments of the controllable image processing method, and will not be repeated here.

[0124] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method or to run the above-described system.

[0125] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc., may be partitioned and stored in one or more memories.

[0126] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0127] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0128] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described method or to run the above-described system.

[0129] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method based on controllable generation, characterized in that, include: Obtain the conditional signals and information extraction model for the image processing task, and construct a first loss function based on the conditional signals and the information extraction model; Choose a pre-trained diffusion model; The final generated image is obtained by iterating based on the pre-trained diffusion model and the first loss function; specifically including: M1 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter; M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the proposed distribution and an unconditional transition probability distribution, obtaining the sequence Monte Carlo weight equation; including: Let x be the nth sample in the t-th iteration. t The diffusion model is defined as x θ (.), where θ represents the model parameters; Using the aforementioned diffusion model x θ (.) From input x t Obtain the unconditional score function s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t Let be the standard deviation of the noise in the t-th iteration; Calculate the conditional score function s θ (x t ,y)=s θ (x t )-αL(y,x θ (x t ), where α is the optimization step size, L is the constructed loss function, and y is the conditional signal; Calculate the likelihood probability distribution p(y|x) θ (x t ))=exp(-L(y,x θ (x t ))) / Z, where exp represents the exponential function and Z is a normalization constant greater than 0; Based on conditional scoring function s θ (x t ,y), establish an optimal estimate of the proposed distribution. and the unconditional transition probability distribution Where N represents a Gaussian distribution and I represents the identity matrix; Optimization estimation of r(x) based on the proposed distribution t-1 |x t and the unconditional transition probability distribution p(x) t-1 |x t The weight equations for sequence Monte Carlo are obtained. M3, based on the Monte Carlo weighting equation and resampling strategy of the sequence, performs resampling to obtain new samples; Repeat steps M1-M3 until the set number of iterations is reached to obtain the final weight equation and the final sampled multiple samples, resulting in the final generated image.

2. The image processing method based on controllable generation according to claim 1, characterized in that, The information extraction model is a noise-reducing image information extraction model. Or conditional information extraction model The conditional signals and information extraction model for acquiring the image processing task are specifically any of the following: - When the image processing task is text-conditional image generation: the conditional signal y is the text condition, the information extraction model H is a CLIP model, and the corresponding denoised image information extraction model... Conditional information extraction model for CLIP image encoder For CLIP text encoder; - When the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, the information extraction model H is the image segmentation network, and the corresponding denoising image information extraction model is... For a pre-trained MobileNetV3-Large model with a segmentation head, the conditional information extraction model This is an identity operation; - When the image processing task is face image conditional image generation: the conditional signal y is the face image condition, the information extraction model H is the face ID feature extraction network, and the corresponding denoising image information extraction model is... For face ID recognition networks, conditional information extraction networks For the same face ID recognition network; - When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, the information extraction model H is the CLIP image encoder, and the corresponding denoised image information extraction model... To obtain the Gram matrix of features for the CLIP image encoder, a conditional information extraction network is used. The same CLIP image encoder obtains the Gram matrix of features.

3. The image processing method based on controllable generation according to claim 1, characterized in that, A first loss function is constructed based on the conditional signal and the information extraction model, specifically one of the following: - When the image processing task is text-conditional image generation: the conditional signal y is the text condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the cosine similarity between y and x0; - When the image processing task is segmentation map conditional image generation: the conditional signal y is the segmentation map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is the sum of the cross-entropy loss of each pixel between the predicted segmentation maps of y and x0. - When the image processing task is face image conditional image generation: the conditional signal y is the face image condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the face ID features of y and x0; - When the image processing task is style map conditional image generation: the conditional signal y is the style map condition, x0 is defined as the predicted original image obtained by the diffusion model, and the loss function L(y,x0) is constructed as the L2 Euclidean distance between the Gram matrices of y and x0.

4. The image processing method based on controllable generation according to claim 1, characterized in that, The sampling of multiple samples from the initial distribution of the sequence Monte Carlo as a particle filter includes: Define T as the set total number of iterations, with a mean of 0 and a variance of T. 2 The Gaussian distribution is the initial distribution of the sequence Monte Carlo. N initial samples are sampled from this initial distribution and used as particle filters.

5. The image processing method based on controllable generation according to claim 1, characterized in that, Based on the weighting equation and resampling strategy of the sequence Monte Carlo method, resampling is performed to obtain new samples, including: Using a sequence Monte Carlo resampling strategy, N samples are resampled based on the N samples and their corresponding weight equations to obtain N new samples.

6. An image processing system based on controllable generation, characterized in that, include: Data acquisition module: Acquires the conditional signal y and information extraction model H for the image processing task; Construction module: Based on the conditional signal and information extraction model obtained by the data acquisition module, a first loss function is constructed. At the same time, a diffusion model is constructed and trained on the image dataset using a second loss function for predicting noise to obtain a pre-trained diffusion model. Image processing module: Iterates based on the pre-trained diffusion model and the first loss function to obtain the final generated image; The image processing module includes: Module M1 samples multiple samples from the initial Monte Carlo distribution of the sequence as a particle filter; Module M2, based on the pre-trained diffusion model and the first loss function, establishes an optimized estimate of the proposed distribution and an unconditional transition probability distribution, obtaining the sequence Monte Carlo weight equation; including: Let x be the nth sample in the t-th iteration. t The diffusion model is defined as x θ (.), where θ represents the model parameters; Using the aforementioned diffusion model x θ (.) From input x t Obtain the unconditional score function s θ (x t )=(x θ (x t )-x t ) / σ t , where σ t Let be the standard deviation of the noise in the t-th iteration; Calculate the conditional score function s θ (x t ,y)=s θ (x t )-αL(y,x θ (x t ), where α is the optimization step size, L is the constructed loss function, and y is the conditional signal; Calculate the likelihood probability distribution p(y|x) θ (x t ))=exp(-L(y,x θ (x t ))) / Z, where exp represents the exponential function and Z is a normalization constant greater than 0; Based on conditional scoring function s θ (x t ,y), establish an optimal estimate of the proposed distribution. and the unconditional transition probability distribution Where N represents a Gaussian distribution and I represents the identity matrix; Optimization estimation of r(x) based on the proposed distribution t-1 |x t and the unconditional transition probability distribution p(x) t-1 |x t The weight equations for sequence Monte Carlo are obtained. Module M3, based on the weighting equation and resampling strategy of the sequence Monte Carlo, performs resampling to obtain new samples; The modules M1-M3 are executed sequentially until the set number of iterations is reached, resulting in the final weight equation and multiple samples, thus generating the final image.

7. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-5, or to run the system of claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-5, or to run the system of claim 6.

Images