Zero sample color style migration method and system based on pre-training diffusion model

Abstract

The invention discloses a zero-sample color style migration method and system based on a pre-training diffusion model, and is applied to the technical field of data processing, and the method comprises the steps: obtaining an original image and a reference image; respectively carrying out inversion on the original image and the reference image to obtain a first noise latent variable and a second noise latent variable; inputting the first noise latent variable and the second noise latent variable into a noise latent variable space for distribution transformation, iteratively updating the noise latent variable through a diffusion model, and outputting a latent variable corresponding to a result image; wherein in the diffusion model, the acquired random noise latent variable and the style text prompt are input into the diffusion model, and an image with a text description style is generated.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a zero-sample color style transfer method and system based on a pre-trained diffusion model. Background Art

[0002] The diffusion model is a type of probability-based generative model. Its basic principle is to destroy the data structure by gradually adding noise, and then learn the reverse process to remove the noise, thereby generating high-quality data. The key concepts of the diffusion model include forward diffusion process, reverse diffusion process, probability distribution, Markov chain, and variational inference. In the fields of image generation, image restoration, super-resolution, and text generation, the diffusion model has achieved remarkable results and has become one of the important technologies in generative artificial intelligence.

[0003] In recent years, diffusion models have shown great potential in the field of image generation and editing. Their unique mechanism of gradual noise addition and removal provides a new idea for color style transfer. However, most existing diffusion models focus on the transfer of artistic styles, and there are relatively few studies on color style transfer, especially color style transfer based on text references.

[0004] In order to overcome these defects, this application proposes a zero-sample color style transfer method and system based on a pre-trained diffusion model. Summary of the invention

[0005] The purpose of this application is to provide a zero-sample color style transfer method and system based on a pre-trained diffusion model, aiming to solve the above-mentioned problems.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides a zero-sample color style transfer method based on a pre-trained diffusion model, comprising:

[0008] Obtaining an original image and a reference image;

[0009] Inverting the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable;

[0010] The first noise latent variable and the second noise latent variable are input into the noise latent variable space for distribution transformation, the noise latent variables are iteratively updated through the diffusion model, and the latent variables corresponding to the result image are output; wherein, the diffusion model also includes: inputting the obtained random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

[0011] In a second aspect, the present application provides a zero-shot color style transfer system based on a pre-trained diffusion model, comprising:

[0012] Acquisition module: acquire original image and reference image;

[0013] Processing and output module: invert the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable; input the first noise latent variable and the second noise latent variable into the noise latent variable space for distribution transformation, iteratively update the noise latent variables through a diffusion model, and output the latent variables corresponding to the result image; wherein, the diffusion model also includes: inputting the obtained random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

[0014] In a third aspect, the present application provides a device, comprising a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a zero-sample color style transfer method based on a pre-trained diffusion model; and the processor is used to execute the program instructions stored in the memory to implement zero-sample color style transfer based on a pre-trained diffusion model.

[0015] In a fourth aspect, the present application provides a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute a zero-sample color style transfer method based on a pre-trained diffusion model.

[0016] This application provides a zero-sample color style transfer method and system based on a pre-trained diffusion model, which has the following beneficial effects:

[0017] (1) Through the DDIM inversion technology, the original image and the reference image are accurately mapped to the noise latent variable space, ensuring the integrity and authenticity of the image content and avoiding the image distortion and content destruction problems caused by the encoder-decoder structure in traditional methods;

[0018] (2) Distribution transformation is performed in the noise latent variable space, and the distribution of the original image and the reference image are aligned using the WCT operation, thus achieving accurate color style migration. Since the distribution transformation is not performed inside the neural network, the problem of mismatch between content and style is avoided.

[0019] (3) By reconstructing the stylized image through DDIM sampling, not only the content consistency of the original image is maintained, but also the perfect migration of color style is achieved; at the same time, this application does not require training or parameter adjustment, and has extremely high efficiency and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1This is a flow chart of a zero-sample color style transfer method based on a pre-trained diffusion model according to Example 1 of the present application;

[0021] Figure 2 This is a flowchart of a zero-sample color style transfer method based on a pre-trained diffusion model according to Example 1 of the present application;

[0022] Figure 3 This is a structural schematic diagram of a zero-sample color style transfer system based on a pre-trained diffusion model according to Example 2 of the present application;

[0023] Figure 4 This is a schematic diagram of the device structure of Example 3 of the present application;

[0024] Figure 5 This is a schematic diagram of the storage medium structure of Example 4 of the present application. DETAILED DESCRIPTION

[0025] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] The following is an analysis of the solutions and existing problems in the prior art in combination with related technologies.

[0027] Lack of personalization: Due to their predefined characteristics, existing filters and LUTs cannot flexibly adapt to users' diverse aesthetic needs, resulting in limitations in users' choice of color styles and a lack of the ability to adapt to color style migration in open scenarios.

[0028] Image quality issues: Traditional color style transfer methods based on encoder-decoder structures or manually adjusting parameters are prone to introducing distortion artifacts such as texture distortion and destruction of image content, especially when processing images with complex content. This is mainly due to the excessive manipulation of image content by the encoder-decoder method during the color mapping process.

[0029] Inconsistency between content and style: Existing color style transfer methods find it difficult to accurately transfer color style while maintaining the consistency of image content, resulting in the problem of mismatch between style and content in the generated results.

[0030] Limitations of text references: Currently, there is little research on color style transfer based on text references. Most existing methods are limited to the conversion of artistic styles. However, the application of text references in color style is not yet mature, and it is unable to process open sets of text references, which limits the user's choice and operational flexibility.

[0031] This application uses the pre-trained Stable Diffusion model to perform migration operations directly in the color style space without retraining or adjusting model parameters.

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] Example 1

[0034] See also Figure 1 , is a flowchart of a zero-sample color style transfer method based on a pre-trained diffusion model according to Example 1 of the present application; the steps include:

[0035] S1: Get the original image and the reference image.

[0036] S2: Invert the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable.

[0037] In this embodiment, the original image I0 and the reference image I r DDIM inversion is performed separately.

[0038] The original image I0 and the reference image I r Inversion to the noise latent variable space Z T , get the first noise latent variable Z T,0 and the second noise latent variable Z T,r The two noise latent variables can be based on the noise latent variable space Z T Sampling is performed to reconstruct I0 and I r .

[0039] It can be understood that DDIM is based on the forward and reverse processes in the diffusion model. By reparameterizing the denoising process, a deterministic sampling path is derived, avoiding the randomness in traditional diffusion sampling.

[0040] S3: Inputting the first noise latent variable and the second noise latent variable into the noise latent variable space for distribution transformation, iteratively updating the noise latent variables through the diffusion model, and outputting the latent variables corresponding to the result image; wherein, the diffusion model also includes: inputting the acquired random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

[0041] In this embodiment, the first noise latent variable and the second noise latent variable are sampled in parallel in the DDIM sampling phase. In the diffusion model, the reverse sampling formula is:

[0042]

[0043] Where z is random noise; α t is the noise attenuation factor; β t is the noise scaling factor at time step t; σ t is the standard deviation of the noise at time step t.

[0044] DDIM removes random noise by redesigning the transformation formula between time steps:

[0045]

[0046] Among them, x t-1 is the latent variable at time step t-1, that is, the latent variable x at the current time step t t The intermediate result generated by denoising; is the cumulative noise reduction factor; is the current latent variable x t and the original image signal reconstructed from the predicted noise; is the weight of the noise in time step t; ∈θ(x t ,t) is the diffusion model for the current latent variable x t and the noise predicted at time step t. There is no need to add random noise z, thus achieving deterministic sampling.

[0047] When the sampling is at time step t, the original image I0 and the reference image I r The intermediate latent variables are and The intermediate latent variable and Input the diffusion model and output the predicted noise. The calculation formula is:

[0048]

[0049] Among them, among them, ∈ o The original image corresponds to The predicted first prediction noise; ∈ θ is the noise prediction function; t is the time step in the sampling process; c o Embedding information for the first text; ∈ r The reference image corresponds to The predicted second prediction noise; c r Embed information for the second text.

[0050] The distribution transformation is defined as:

[0051]

[0052] in, To operate through WCT will ∈ o and ∈ r The result after the distribution alignment, WCT is the weighted color transform.

[0053] WCT operations include:

[0054] Extracting features of the original image and the reference image through a pre-trained convolutional neural network;

[0055] Remove the distribution correlation of the obtained features to obtain features with zero mean, unit variance and independence;

[0056] The feature matrix F of the original image s To do this:

[0057] Calculate the covariance matrix

[0058] By eigendecomposition

[0059] The feature matrix of the original image after bleaching is:

[0060]

[0061] Among them, F s is the feature matrix of the original image; F s The transposed matrix of s is the covariance matrix of the feature matrix of the original image; n is the number of features, that is, F s The number of columns; U s is the orthogonal eigenvector of the feature matrix of the original image; s is the first diagonal matrix, and the elements on the diagonal are the eigenvalues ​​of the covariance matrix; For U s The transposed matrix of

[0062] Re-associate the bleached feature matrix to the feature distribution of the reference image;

[0063] The feature matrix F of the reference image t To do this:

[0064] Calculate the covariance matrix

[0065] By eigendecomposition

[0066] The feature matrix of the original image after bleaching is colored by the feature distribution of the reference image:

[0067]

[0068] Among them, U t is the orthogonal eigenvector of the feature matrix of the reference image; t is the second diagonal matrix; For U t The transposed matrix of

[0069] The features transformed by WCT are reconstructed into an image through the inverse operation of the convolutional network.

[0070] The noise latent variable is updated iteratively through the diffusion model, and the update formula is:

[0071]

[0072] Among them, ∈ t is the noise updated in time step t; w is the weight parameter; ∈ u is the unconditional noise of the prediction, i.e., the noise of the diffusion model output without the preset condition information; is the intermediate latent variable updated from time step t to t-1 through the Reverse operation; is the intermediate latent variable at time step t;

[0073] After T iterations, the latent variable corresponding to the result image is obtained

[0074] Furthermore, it also includes color style transfer based on text reference, specifically: sampling random noise latent variable z based on standard normal distribution N(0,1) T,r , z T,r and get the style text hint c s Input to the noise prediction function of the diffusion model.

[0075] When the sampling is at time step t, the intermediate latent variables corresponding to the original image and the style text prompt are generated based on the random noise latent variable: and The noise is predicted by the diffusion model output; the calculation formula is:

[0076]

[0077] Among them, W o At time step t, the latent variable corresponding to the original image The predicted noise; ∈ θ is the noise prediction function; y o is the condition information related to the original image, such as the content or features of the original image; W r is the latent variable corresponding to the style text prompt at time step t The predicted noise; cs is the style text prompt, such as the style description in the text prompt; θ is the parameter of the diffusion model.

[0078] After T iterations, the latent variables corresponding to the image of the text description style are obtained.

[0079] In summary, in Example 1 of the present application, DDIM (inversion) is performed on the original image and the reference image, and they are mapped to the noise latent variable space to obtain the corresponding noise latent variable; in the noise latent variable space, the distribution of the original image and the reference image is aligned by the WCT operation to achieve the transformation of the style distribution; in this process, a parallel sampling mechanism is adopted to ensure that the source image and the reference image maintain content consistency while the style is transferred; through the DDIM sampling method, a stylized image is reconstructed from the transformed noise latent variable to complete the color style transfer. The present application does not require complex model training or parameter adjustment, and realizes efficient style transfer with zero samples and zero parameter adjustment, providing users with a flexible and personalized image editing experience.

[0080] Example 2

[0081] See also Figure 3 , which is a structural diagram of a zero-sample color style transfer system based on a pre-trained diffusion model according to Example 2 of the present application; the specific contents include:

[0082] Acquisition module: acquire original image and reference image;

[0083] Processing and output module: invert the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable; input the first noise latent variable and the second noise latent variable into the noise latent variable space for distribution transformation, iteratively update the noise latent variables through a diffusion model, and output the latent variables corresponding to the result image; wherein, the diffusion model also includes: inputting the obtained random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

[0084] In this embodiment, the system achieves full customization of colors by obtaining user text descriptions or image references. For example, if the text describes "luxurious golden tones" or "warm sunset orange", the system will generate the most matching color style, no longer limited to predefined filters or style libraries.

[0085] Based on the diffusion model, that is, the pre-trained Stable Diffusion model, no complex model training or parameter adjustment is required. This plug-and-play feature will help users save a lot of time and cost while obtaining top-level personalized services. Stable Diffusion is a deep generative model based on the diffusion model, focusing on high-quality image generation and editing tasks, using the reverse simulation of the diffusion process to gradually generate images from noise. The pre-trained Stable Diffusion model has learned rich visual features and semantic relationships by training on large-scale annotated or weakly annotated image datasets, thus having powerful generation capabilities.

[0086] The system can also automatically identify local areas in the image and change the style of the user-specified area through precise local color editing. For example, the user specifies an area in the image, such as "forest", and the system generates the corresponding color. The system model proposed in this application is a groundbreaking zero-sample migration technology that can not only better preserve the details and texture of the original image, but also make color style matching more accurate and natural; it can be applied to various open scenes without the need for additional network parameters.

[0087] Example 3

[0088] See also Figure 4 , is a schematic diagram of the device structure of Embodiment 3 of the present application. The device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

[0089] The memory 52 stores program instructions for implementing the above-mentioned zero-sample color style transfer method based on the pre-trained diffusion model.

[0090] The processor 51 is used to execute program instructions stored in the memory 52 to implement zero-sample color style transfer based on the pre-trained diffusion model.

[0091] The processor 51 may also be referred to as a CPU (Central Processing Unit).

[0092] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] Example 4

[0094] See also Figure 5, which is a schematic diagram of the structure of the storage medium of Example 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or a computer, a server, a mobile phone, a tablet, and other devices.

[0095] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0096] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

[0097] Although the embodiments of the present application have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the appended claims and their equivalents.

[0098] Of course, the present invention may have many other implementations. Based on this implementation, other implementations obtained by ordinary technicians in this field without any creative work are all within the scope of protection of the present invention.

Claims

1. A zero-shot color style transfer method based on a pre-trained diffusion model, characterized in that: include: Obtaining an original image and a reference image; Inverting the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable; The first noise latent variable and the second noise latent variable are input into the noise latent variable space for distribution transformation, the noise latent variables are iteratively updated through the diffusion model, and the latent variables corresponding to the result image are output; wherein, the diffusion model also includes: inputting the obtained random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

2. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 1, characterized in that: The step of respectively inverting the original image and the reference image to obtain the first noise latent variable and the second noise latent variable specifically includes the following steps: The original image I0 and the reference image I r DDIM inversion was performed separately; The original image I0 and the reference image I r Inversion to the noise latent variable space Z T , get the first noise latent variable Z T,0 and the second noise latent variable Z T,r ; Based on the noise latent variable space Z T Sampling is performed to reconstruct I0 and I r .

3. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 1, characterized in that: The step of inputting the first noise latent variable and the second noise latent variable into the noise latent variable space for distribution transformation, iteratively updating the noise latent variable through the diffusion model, and outputting the latent variable corresponding to the result image specifically includes the following steps: In the DDIM sampling phase, the first noise latent variable and the second noise latent variable are sampled in parallel; When the sampling is at time step t, the original image I0 and the reference image I r The intermediate latent variables are and The intermediate latent variable and Input the diffusion model and output the predicted noise. The calculation formula is: Among them, ∈ o The original image corresponds to The predicted first prediction noise; ∈ θ is the noise prediction function; t is the time step in the sampling process; c o Embedding information for the first text; ∈ r The reference image corresponds to The predicted second prediction noise; ∈ r embedding information for the second text; The distribution transformation is defined as: in, To operate through WCT will ∈ o and ∈ r The result after the distribution alignment; WCT is the weighted color transform.

4. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 3, characterized in that: The iterative updating of the noise latent variable through the diffusion model includes: The update formula is: Among them, ∈ t is the noise updated in time step t; w is the weight parameter; ∈ u is the unconditional noise of the prediction, i.e., the noise of the diffusion model output without the preset condition information; is the intermediate latent variable updated from time step t to t-1 through the Reverse operation; is the intermediate latent variable at time step t; After T iterations, the latent variable corresponding to the result image is obtained 5. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 3, characterized in that: The WCT operation includes: Extracting features of the original image and the reference image through a pre-trained convolutional neural network; Remove the distribution correlation of the obtained features to obtain features with zero mean, unit variance and independence; The feature matrix F of the original image s To do this: Calculate the covariance matrix By eigendecomposition The feature matrix of the original image after bleaching is: Among them, F s is the feature matrix of the original image; F s The transposed matrix of s is the covariance matrix of the feature matrix of the original image; n is the number of features, that is, F s The number of columns; U s and Λ s is the covariance matrix C s The characteristic decomposition result, U s is the orthogonal eigenvector of the feature matrix of the original image, Λ s is the first diagonal matrix, and the elements on the diagonal are the eigenvalues ​​of the covariance matrix; For U s The transposed matrix of Re-associate the bleached feature matrix to the feature distribution of the reference image; The feature matrix F of the reference image t To do this: Calculate the covariance matrix By eigendecomposition The feature matrix of the original image after bleaching is colored by the feature distribution of the reference image: Among them, U t and Λ t is the covariance matrix C t The characteristic decomposition result, U t is the orthogonal eigenvector of the feature matrix of the reference image; t is the second diagonal matrix; For U t The transposed matrix of The features transformed by WCT are reconstructed into an image through the inverse operation of the convolutional network.

6. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 1, characterized in that: The step of inputting the acquired random noise latent variables and the style text prompt into the diffusion model to generate an image of the text description style specifically includes the following steps: Sampling random noise latent variable z based on standard normal distribution N(0,1) T,r , z T,r and get the style text hint c s The noise prediction function of the diffusion model is input, and the predicted noise is output; after T iterations, the latent variable corresponding to the image of the text description style is obtained.

7. The zero-shot color style transfer method based on a pre-trained diffusion model according to claim 1, characterized in that: The diffusion model is a Stable Diffusion model; In the diffusion model, DDIM is based on the forward and reverse processes of the diffusion model; The reverse sampling formula is: Where z is random noise; α t is the noise attenuation factor; β t is the noise scaling factor at time step t; σ t is the standard deviation of the noise at time step t; DDIM removes random noise by redesigning the transformation formula between time steps. The formula is: Among them, x t-1 is the latent variable at time step t-1, that is, the latent variable x at the current time step t t The intermediate result generated by denoising; is the cumulative noise reduction factor; is the current latent variable x t and the original image signal reconstructed from the predicted noise; is the weight of the noise in time step t; ∈θ(x t ,t) is the diffusion model for the current latent variable x t and the noise predicted at time step t.

8. A system for the zero-shot color style transfer method based on a pre-trained diffusion model according to claim 1, characterized in that: include: Acquisition module: acquire original image and reference image; Processing and output module: inverting the original image and the reference image respectively to obtain a first noise latent variable and a second noise latent variable; The first noise latent variable and the second noise latent variable are input into the noise latent variable space for distribution transformation, the noise latent variables are iteratively updated through the diffusion model, and the latent variables corresponding to the result image are output; wherein, the diffusion model also includes: inputting the obtained random noise latent variables and style text prompts into the diffusion model to generate an image of the text description style.

9. A device, characterized in that: The device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing the zero-sample color style transfer method based on a pre-trained diffusion model as described in any one of claims 1-7; and the processor is used to execute the program instructions stored in the memory to implement zero-sample color style transfer based on a pre-trained diffusion model.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute the zero-sample color style transfer method based on a pre-trained diffusion model according to any one of claims 1 to 7.

Images