Text-driven image-to-image conversion method based on wavelet transform band fusion

Abstract

The application discloses a text-driven image-to-image conversion method based on wavelet transform band fusion, and belongs to the technical field of image processing and deep learning; the method comprises the following steps: obtaining noise coding and semantic-guided random noise of a reference image; fusing the noise coding and the semantic-guided random noise through two-dimensional discrete wavelet transform to generate semantic-guided random noise of a fused reference image structure; iteratively denoising and multi-scale fusing the noise coding and the semantic-guided random noise of the fused reference image structure; after a preset wavelet transform time step, denoising the semantic-guided random noise for multiple times until a total denoising time step is completed, so that final noise combined with the reference image structure and text prompt information is obtained; and restoring the final noise to a pixel space to output a reconstructed image; the application can accurately retain structure information of the reference image.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technology, and in particular to a text-driven image-to-image conversion method based on wavelet transform frequency band fusion. Background Technology

[0002] Text-driven image-to-image translation technology has received widespread attention in the field of image generation in recent years. Its core task is to achieve semantic-level content transformation based on natural language descriptions while maintaining the basic structure of the reference image. Existing methods mainly face the challenge of effectively coordinating structure and semantics.

[0003] Current mainstream methods suffer from the following technical limitations: First, attention-based feature fusion methods (such as Plug-and-Play) typically perform global fusion in the feature space, failing to fully consider the frequency domain characteristics where image structural information is mainly distributed in low-frequency components and detailed textures exist in high-frequency components. This can easily damage the key structural information of the reference image during semantic editing. Second, while frequency domain methods based on discrete cosine transform can separate frequency components to some extent, their ability to preserve edge structures and directional features is limited, making them prone to structural distortion when processing complex contours. Furthermore, existing methods lack refined weight adjustment mechanisms, making it difficult to achieve a flexible balance between preserving the reference image structure and adhering to textual semantics. Users find it difficult to adjust the strength of structure preservation according to specific needs.

[0004] In particular, in image editing methods operating in noisy spaces, effectively injecting structural information from reference images into the semantically guided generation process remains a technical challenge. Direct noise fusion easily leads to the loss of structural information, while simple feature concatenation causes the generated result to deviate from the text's semantics. Therefore, a novel technical solution is needed that can process structural and semantic information separately in different frequency subbands and support controllable fusion. Summary of the Invention

[0005] The purpose of this invention is to provide a text-driven image-to-image conversion method based on wavelet transform frequency band fusion, which can accurately preserve the structural information of the reference image.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a text-driven image-to-image conversion method based on wavelet transform frequency band fusion, comprising the following steps:

[0007] Obtain the noise encoding of the reference image and the embedding vector of the text prompt; randomly generate multiple noises with the same size as the noise encoding, and fuse them with the embedding vector of the text prompt to generate semantically guided random noise;

[0008] The noise coding and semantically guided random noise are processed through two-dimensional discrete wavelet transform decomposition, weighted sub-band fusion, and inverse transform reconstruction to generate semantically guided random noise that fuses the reference image structure. Specifically, the two-dimensional discrete wavelet transform decomposition decomposes the noise coding and semantically guided random noise into four wavelet sub-bands: one low-frequency sub-band contains the main structural and stylistic information of the reference image, and three high-frequency wavelet sub-bands retain the edge information of the reference image in the horizontal, vertical, and diagonal directions, respectively. The weighted sub-band fusion is based on the relative weights of the reference image and the text prompt, performing frequency band fusion on the wavelet sub-bands corresponding to the noise coding and semantically guided random noise. The inverse transform reconstruction reconstructs the fused wavelet sub-bands using two-dimensional discrete wavelet inverse transform to obtain the semantically guided random noise that fuses the reference image structure.

[0009] The semantically guided random noise of the noise coding and the fused reference image structure is denoised. The denoised noise coding and the semantically guided random noise of the fused reference image structure are then processed again by two-dimensional discrete wavelet transform decomposition, weighted subband fusion and inverse transform reconstruction to generate the semantically guided random noise of the denoised fused reference image structure, which is used for the next round of denoising iteration.

[0010] After setting the time step for wavelet transform denoising, the output semantically guided random noise is denoised multiple times until the total denoising time step is completed, resulting in the final noise that combines the reference image structure and text prompt information; the final noise is then restored to the pixel space, and the reconstructed image is output.

[0011] In some optional embodiments, the weighted subband fusion formula is as follows:

[0012] ;

[0013] In the formula, , , and The four wavelet subbands of the noise coding decomposition are, among which Includes key structural and stylistic information from the reference image. , and The three high-frequency sub-bands respectively retain the edge information of the reference image in the horizontal, vertical and diagonal directions; , , and Four wavelet subbands for semantically guided random noise decomposition; For each subband in noise coding, the weight parameters are: when When the value approaches 1, the generated image retains more structural information from the reference image; when... When the value approaches 0, the generated image better reflects the semantics of the text.

[0014] In some optional embodiments, obtaining the noise encoding of the reference image and the embedding vector of the text prompt includes the following steps:

[0015] The reference image is transformed into a latent space vector using the encoder in a variational autoencoder (VAE). Noise is then progressively added to the latent space vector using the DDIMInversion technique to obtain a noise code containing information from the reference image. The formula for progressively adding noise is as follows:

[0016] ;

[0017] In the formula, This is the latent space vector after adding noise for time step t+1; The latent space vector before noise addition corresponds to time step t; For denoising U-Net networks; No text input; In time step The noise scheduling coefficient at each time step is used to control the noise input during the forward process;

[0018] Text prompt information is encoded into text prompt embedding vectors by a text encoder that is pre-trained with CLIP using a contrastive language-image model.

[0019] In some optional embodiments, the denoising of semantically guided random noise in the noise coding and fusion reference image structure includes the following steps:

[0020] use The reverse sampling loop of step denoising progressively denoises the noise code, and the formula for progressive denoising of the noise code is as follows:

[0021] ;

[0022] In the formula, This is the denoised latent space vector corresponding to time step t-1; The latent space vector before denoising corresponds to time step t; In time step Noise scheduling coefficients at time -1 and t; To introduce the intensity of noise, when The sampling process becomes a deterministic process; when The randomness of the sampling process increases; in order to ensure that the reconstructed image is consistent with the reference image, let ; No text input; For denoising U-Net networks; It is random noise.

[0023] The classifier-free guided CFG technique is used to progressively denoise the guided random noise. The predicted noise is corrected using the following formula:

[0024] ;

[0025] ;

[0026] In the formula, The noise prediction results are used for subsequent processes; To control the guiding strength of CFG, the value is greater than 1; To balance between no text input and irrelevant text input, the value ranges from [0,1]. For denoising U-Net networks; No text input; P represents the target text; Input text that is irrelevant to the target; Let be the noisy latent variable at the current time step t.

[0027] In some optional embodiments, the step of restoring the final noise to pixel space and outputting the reconstructed image specifically includes:

[0028] The decoder in the Variational Autoencoder (VAE) is used to restore the final noise to pixel space, outputting the reconstructed image.

[0029] The text-driven image-to-image conversion method based on wavelet transform frequency band fusion provided by this invention has at least the following beneficial effects:

[0030] This invention obtains the noise encoding of a reference image and the embedding vector of the text prompt, generates semantically guided random noise, and decomposes the noise encoding and semantically guided random noise into wavelet subbands containing structural and edge information using two-dimensional discrete wavelet transform. It sets the relative weights of the reference image and the text prompt and achieves subband frequency fusion through two-dimensional discrete wavelet inverse transform. The invention also presets the wavelet transform time step and the total denoising time step, repeating the fusion and denoising process until a reconstructed image is output. This technical solution accurately preserves the structure of the reference image, achieving controllable fusion of structure and semantics without model fine-tuning. Furthermore, a closed-loop iterative process ensures stable and reliable generation results, effectively solving the core problems of existing text-driven image-to-image conversion technologies, such as easily destroyed structure, uncontrollable fusion strength, reliance on fine-tuning, and high computational power. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0032] Figure 1This is a flowchart of a text-driven image-to-image conversion method based on wavelet transform frequency band fusion according to an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the detailed calculation process of frequency band fusion according to an embodiment of the present invention;

[0034] Figure 3 This is an engine block diagram of a text-driven image-to-image conversion method based on wavelet transform frequency band fusion according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0036] One embodiment of the present invention relates to a text-driven image-to-image conversion method based on wavelet transform frequency band fusion. The implementation details of the text-driven image-to-image conversion method based on wavelet transform frequency band fusion in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0037] The specific process of the text-driven image-to-image conversion method based on wavelet transform frequency band fusion in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0038] Step 101: Obtain the noise encoding of the reference image and the embedding vector of the text prompt; randomly generate multiple noises with the same size as the noise encoding, and fuse them with the embedding vector of the text prompt to generate semantically guided random noise;

[0039] The reference image is transformed into a latent space vector using the encoder in a variational autoencoder (VAE). Noise is then progressively added to the latent space vector using the DDIMInversion technique to obtain a noise code containing information from the reference image. The formula for progressively adding noise is as follows:

[0040] ;

[0041] In the formula, This is the latent space vector after adding noise for time step t+1; The latent space vector before noise addition corresponds to time step t; For denoising U-Net networks; No text input; In time step The noise scheduling coefficient at time step is used to control the noise input during the forward process.

[0042] Text prompts are encoded into embedding vectors using a text encoder that is pre-trained with CLIP based on contrastive language and images.

[0043] Step 102 involves processing the noise coding and semantically guided random noise through two-dimensional discrete wavelet transform decomposition, weighted sub-band fusion, and inverse transform reconstruction to generate semantically guided random noise that fuses the reference image structure. Specifically, the two-dimensional discrete wavelet transform decomposition decomposes the noise coding and semantically guided random noise into four wavelet sub-bands: one low-frequency sub-band contains the main structural and stylistic information of the reference image, and three high-frequency wavelet sub-bands retain the edge information of the reference image in the horizontal, vertical, and diagonal directions, respectively. The weighted sub-band fusion is based on the relative weights of the reference image and the text prompt, performing frequency band fusion on the wavelet sub-bands corresponding to the noise coding and semantically guided random noise. The inverse transform reconstruction involves reconstructing the fused wavelet sub-bands using two-dimensional discrete wavelet inverse transform to obtain the semantically guided random noise that fuses the reference image structure.

[0044] The frequency band fusion formula based on wavelet transform is as follows:

[0045] ;

[0046] In the formula, , , and The four wavelet subbands of the noise coding decomposition are, among which Includes key structural and stylistic information from the reference image. , and The three high-frequency sub-bands respectively retain the edge information of the reference image in the horizontal, vertical and diagonal directions; , , and Four wavelet subbands for semantically guided random noise decomposition; For each subband in noise coding, the weight parameters are: when When the value approaches 1, the generated image retains more structural information from the reference image; when... When the value approaches 0, the generated image better reflects the semantics of the text.

[0047] The detailed calculation process of frequency band fusion is as follows: Figure 2 As shown in the figure, the feature fusion process based on two-dimensional discrete wavelet transform (2D-DWT) is illustrated: first, the content branch features are processed... and semantic branch features Perform 2D-DWT separately to decompose into low-frequency (ll), horizontal high-frequency (lh), vertical high-frequency (hl), and diagonal high-frequency (hh) subbands; then apply the weight matrix. The content branch sub-band weights are 1 - The corresponding subbands are weighted (e.g., the semantic branch subband weights), and then the fused semantic branch features are obtained by performing a two-dimensional discrete wavelet inverse transform (2D-IDWT). This allows for the controllable integration of content structure and semantic information.

[0048] Step 103: Denoise the semantically guided random noise of the noise coding and the fusion reference image structure. Then, denoise the semantically guided random noise of the noise coding and the fusion reference image structure again through two-dimensional discrete wavelet transform decomposition, weighted subband fusion and inverse transform reconstruction to generate the semantically guided random noise of the denoised fusion reference image structure for the next round of denoising iteration.

[0049] use The reverse sampling loop of step denoising progressively denoises the noise code, and the formula for progressive denoising of the noise code is as follows:

[0050] ;

[0051] In the formula, This is the denoised latent space vector corresponding to time step t-1; The latent space vector before denoising corresponds to time step t; In time step Noise scheduling coefficients at time -1 and t; To introduce the intensity of noise, when The sampling process becomes a deterministic process; when The randomness of the sampling process increases; in order to ensure that the reconstructed image is consistent with the reference image, let ; No text input; For denoising U-Net networks; It is random noise.

[0052] The classifier-free guided CFG technique is used to progressively denoise the guided random noise. The predicted noise is corrected using the following formula:

[0053] ;

[0054] ;

[0055] In the formula, The noise prediction results are used for subsequent processes; To control the guiding strength of CFG, the value is greater than 1; To balance between no text input and irrelevant text input, the value ranges from [0,1]. For denoising U-Net networks; No text input; P represents the target text; Input text that is irrelevant to the target; Let be the noisy latent variable at the current time step t.

[0056] Step 104: After the preset wavelet transform denoising time step, the output semantically guided random noise is denoised multiple times until the total denoising time step is completed, and the final noise combining the reference image structure and text prompt information is obtained; the final noise is restored to the pixel space and the reconstructed image is output.

[0057] The decoder in the Variational Autoencoder (VAE) is used to restore the final noise to pixel space, outputting the reconstructed image.

[0058] The engine block diagram of the text-driven image-to-image conversion method based on wavelet transform frequency band fusion is as follows: Figure 3 As shown in the figure, two parallel processes of image conversion are presented: the upper part is the reconstruction process of the content branch, in which the input image is encoded, the DDIM inverse process is performed, and wavelet transform is combined with other operations to finally reconstruct the image; the lower part is the conversion process of the semantic branch, which is based on random noise and, with the text prompt "a man", uses wavelet transform in collaboration with the content branch to generate a converted image that conforms to the semantics of the text through multiple steps.

[0059] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0060] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0061] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A text-driven image-to-image conversion method based on wavelet transform frequency band fusion, characterized in that, The method includes: Obtain the noise encoding and text prompt embedding vectors of the reference image; randomly generate multiple noises with the same size as the noise encoding, and fuse them with the text prompt embedding vectors to generate semantically guided random noise; The noise coding and semantically guided random noise are processed through two-dimensional discrete wavelet transform decomposition, weighted sub-band fusion, and inverse transform reconstruction to generate semantically guided random noise that fuses the reference image structure. Specifically, the two-dimensional discrete wavelet transform decomposition decomposes the noise coding and semantically guided random noise into four wavelet sub-bands: one low-frequency sub-band contains the main structural and stylistic information of the reference image, and three high-frequency wavelet sub-bands retain the edge information of the reference image in the horizontal, vertical, and diagonal directions, respectively. The weighted sub-band fusion performs frequency band fusion on the wavelet sub-bands corresponding to the noise coding and semantically guided random noise based on the relative weights of the reference image and the text prompt. The inverse transform reconstruction reconstructs the fused wavelet sub-bands using two-dimensional discrete wavelet inverse transform to obtain the semantically guided random noise that fuses the reference image structure. The semantically guided random noise of the noise coding and the fused reference image structure is denoised. The denoised noise coding and the semantically guided random noise of the fused reference image structure are then processed again by two-dimensional discrete wavelet transform decomposition, weighted subband fusion and inverse transform reconstruction to generate the semantically guided random noise of the denoised fused reference image structure, which is used for the next round of denoising iteration. After setting the time step for wavelet transform denoising, the output semantically guided random noise is denoised multiple times until the total denoising time step is completed, resulting in the final noise that combines the reference image structure and text prompt information; the final noise is then restored to the pixel space, and the reconstructed image is output. The weighted subband fusion formula is as follows: ； In the formula, , , and The four wavelet subbands of the noise coding decomposition are, among which Includes key structural and stylistic information from the reference image. , and The three high-frequency sub-bands respectively retain the edge information of the reference image in the horizontal, vertical and diagonal directions; , , and Four wavelet subbands for semantically guided random noise decomposition; For each subband in noise coding, the weight parameters are: when When the value approaches 1, the generated image retains more structural information from the reference image; when... Approaching 0, the generated image better reflects the semantics of the text; The denoising of semantically guided random noise in the noise coding and fusion reference image structure includes the following steps: use The reverse sampling loop of step denoising progressively denoises the noise code, and the formula for progressive denoising of the noise code is as follows: ； In the formula, This is the denoised latent space vector corresponding to time step t-1; The latent space vector before denoising corresponds to time step t; In time step Noise scheduling coefficients at time -1 and t; To introduce the intensity of noise, when The sampling process becomes a deterministic process; when The randomness of the sampling process increases; in order to ensure that the reconstructed image is consistent with the reference image, let ; No text input; For denoising U-Net networks; It is random noise; The classifier-free guided CFG technique is used to progressively denoise the guided random noise. The predicted noise is corrected using the following formula: ； ； In the formula, The noise prediction results are used for subsequent processes; To control the guiding strength of CFG, the value is greater than 1; To balance between no text input and irrelevant text input, the value ranges from [0,1]. For denoising U-Net networks; No text input; P represents the target text; Input text that is irrelevant to the target; Let t be the latent space vector before denoising, corresponding to time step t.

2. The text-driven image-to-image conversion method based on wavelet transform frequency band fusion as described in claim 1, characterized in that, The process of obtaining the noise encoding of the reference image and the embedding vector of the text prompt includes the following steps: The reference image is transformed into a latent space vector using the encoder in a variational autoencoder (VAE). Noise is then progressively added to the latent space vector using the DDIMInversion technique to obtain a noise code containing information from the reference image. The formula for progressively adding noise is as follows: ； In the formula, This is the latent space vector after adding noise for time step t+1; The latent space vector before denoising corresponds to time step t; For denoising U-Net networks; No text input; In time step The noise scheduling coefficient at each time step is used to control the noise input during the forward process; Text prompt information is encoded into text prompt embedding vectors by a text encoder that is pre-trained with CLIP using a contrastive language-image model.

3. The text-driven image-to-image conversion method based on wavelet transform frequency band fusion as described in claim 1, characterized in that, The step of restoring the final noise to the pixel space and outputting the reconstructed image specifically includes: The decoder in the Variational Autoencoder (VAE) is used to restore the final noise to pixel space, outputting the reconstructed image.

Images