Image editing method and system based on diffusion model latent space frequency domain adaptive selective enhancement

Abstract

This invention relates to an image editing method and system based on latent space frequency domain adaptive selective enhancement using a diffusion model. The image editing method specifically includes: preprocessing the image to be edited and then encoding it using a variational autoencoder based on a diffusion model to obtain a latent representation of the image in the latent space; wherein the four channels of the latent representation include two structure channels and two style channels; automatically adjusting the adaptive editing intensity by analyzing the high-frequency energy proportion of the structure channels; performing frequency domain bandpass enhancement on the style channels based on the adaptive editing intensity to obtain the enhanced latent representation of the style channels; fusing the latent representations of the two structure channels and the two enhanced style channels to obtain the edited latent representation; and then inputting the edited latent representation into a variational autoencoder for decoding after UNet denoising to generate the output image. This invention achieves structure fidelity, content-aware adaptive adjustment, spectral quality assurance, and plug-and-play functionality.

Description

Technical Field

[0001] This invention belongs to the field of generative artificial intelligence image processing technology, specifically relating to an image editing method and system based on diffusion model latent spatial frequency domain adaptive selective enhancement. Background Technology

[0002] Existing image editing and style transfer methods mostly operate in pixel space, resulting in high computational costs, susceptibility to structural detail corruption, and difficulty in decoupling style from structure. While traditional methods based on Generative Adversarial Networks (GANs) can achieve stylization through adversarial training, training is unstable and generalization is limited.

[0003] Diffusion models significantly reduce computational overhead by mapping images to the latent space using a variational autoencoder (VAE). Stable Diffusion, a representative latent diffusion model, uses its VAE encoder to compress a 512×512 RGB image into a 4×64×64 latent representation, with each of the four channels carrying different semantic information: [channels to be filled in]. , Primarily encodes structural information such as edges, contours, and spatial layout; channels , It primarily encodes style information such as color and texture.

[0004] Existing potential space editing methods have the following shortcomings: First, the neglect of channel semantics leads to structural distortion. Existing methods typically apply a uniform transformation to all four channels in the VAE latent space, without distinguishing the semantic functions carried by each channel; when structural channels are analyzed... , When modifications are applied, geometric distortion is inevitably introduced, which manifests as obvious artifacts such as distorted object boundaries and misaligned textures. Second, the editing intensity lacks content awareness. Existing methods rely on manually set global hyperparameters to control the editing intensity, which cannot adaptively adjust according to the complexity of the image content. Images with simple structures (such as smooth backgrounds or single objects) can withstand stronger enhancements, while images with complex structures (such as cluttered scenes or fine textures) require more conservative processing to avoid structural collapse. This "one-size-fits-all" strategy reduces the consistency of editing results and increases the burden of parameter tuning in actual use. Third, frequency domain filtering introduces spectral artifacts. Some methods employ hard threshold filtering (such as ideal low-pass / band-pass filters) in frequency domain operations, which generates Gibbs ringing and block artifacts at spectral boundaries, significantly reducing visual quality in a perceptually significant manner. Summary of the Invention

[0005] Based on the aforementioned shortcomings and deficiencies in the prior art, one of the objectives of this invention is to at least solve one or more of the aforementioned problems in the prior art. In other words, one of the objectives of this invention is to provide an image editing method and system based on diffusion model latent spatial frequency domain adaptive selective enhancement that meets one or more of the aforementioned requirements.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: An image editing method based on diffusion model latent spatial frequency domain adaptive selective enhancement includes the following steps: S1. After preprocessing the image to be edited, it is encoded by a variational autoencoder of a diffusion model to obtain the latent representation of the image in the latent space; wherein, the four channels of the latent representation include two structure channels and two style channels; S2. By analyzing the high-frequency energy ratio of the structural channel, the adaptive editing intensity is automatically adjusted. Based on the adaptive editing intensity, the style channel is enhanced by frequency domain bandpass enhancement to obtain the potential representation of the enhanced style channel. The latent representations of the two structural channels and the two enhanced style channels are fused to obtain the edited latent representation; S3. After denoising the edited latent representation using UNet, input it into the variational autoencoder for decoding to generate the output image.

[0007] As a preferred embodiment, in step S1, the four channels of the latent representation are respectively , , , ; in, , For structural channels, , For style channels.

[0008] As a preferred embodiment, in step S1, the preprocessing includes center cropping and scaling to the target resolution.

[0009] As a preferred embodiment, in step S2, the high-frequency energy ratio of the structural channel... for: ; Where r is the Euclidean distance from any point on the spectral plane to the zero-frequency position of the spectral center. The preset mid-frequency boundary radius, For two-dimensional Fourier transform, This is a potential representation of the structural channel. Indicates the corresponding structural channel , .

[0010] As a preferred embodiment, in step S2, the adaptive editing intensity for: ; in, Based on editing intensity, For the clamp function, , These are the lower and upper limits of the scaling factor. The target is the proportion of high-frequency energy.

[0011] As a preferred embodiment, the frequency domain bandpass enhancement process includes: The latent representation of the style channel is sequentially subjected to a Tuck window function, a Fourier transform, a Gaussian bandpass filter, and a two-dimensional inverse Fourier transform to obtain the enhanced latent representation of the style channel.

[0012] As a preferred embodiment, in step S2, the latent representation of the enhanced style channel for: ; in, As a potential representation of the style channel, Indicates the corresponding style channel , , Injecting coefficients into style, This is a two-dimensional inverse Fourier transform. Let be the frequency domain transfer function of the Gaussian bandpass filter.

[0013] As a preferred embodiment, the edited potential representation for: .

[0014] The present invention also provides an image editing system based on diffusion model latent spatial frequency domain adaptive selective enhancement, applying the image editing method as described in any of the preceding embodiments, wherein the image editing system includes: The preprocessing module is used to preprocess the image to be edited; The latent coding module is used to encode the preprocessed image to obtain the latent representation of the image in the latent space; The adaptive intensity module is used to automatically adjust the adaptive editing intensity by analyzing the proportion of high-frequency energy in the structural channels; The frequency domain editing module is used to perform frequency domain bandpass enhancement on the style channel based on adaptive editing intensity, so as to obtain the potential representation of the enhanced style channel. The pipeline injection module is used to fuse the latent representations of two structural channels and two enhanced style channels to obtain the edited latent representation; it is also used to perform UNet denoising by non-intrusive injection into the UNet denoising network. The decoding output module is used to decode and output the latent representation after UNet denoising to generate the output image.

[0015] Compared with the prior art, the beneficial effects of this invention are: (1) Structural fidelity; This invention fundamentally avoids the intrusion of editing operations on structural channels through a strict channel differentiation strategy. Unlike the method of indirectly protecting the structure through parameter tuning, this invention eliminates structural distortion at the design level. This design eliminates the inherent trade-off between expressiveness and structural integrity in existing methods. (2) Content-aware adaptive adjustment; This invention uses an adaptive editing intensity mechanism to automatically adjust the degree of aggressiveness of editing based on the structural complexity of the image during inference without the need for manual setting of a global fixed intensity; For batch processing scenarios containing hundreds or thousands of heterogeneous images, this mechanism achieves true intelligent adjustment; (3) Spectral quality assurance; This invention uses Gaussian spectrum smooth roll-off instead of hard threshold cutoff, and combines spatial window function to suppress spectral boundary artifacts, enhancing style channels without introducing visual artifacts such as ringing and block partitioning, which are traditional frequency domain methods. (4) Plug and play; This invention is designed without training, without modifying any model weights, and can be directly connected to existing image editing pipelines based on latent diffusion models without additional training or fine-tuning. Attached Figure Description

[0016] Figure 1 This is a flowchart of an image editing method based on diffusion model latent spatial frequency domain adaptive selective enhancement according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the four-channel differentiated editing strategy according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the adaptive intensity adjustment relationship according to an embodiment of the present invention; Figure 4 This is a module architecture diagram of an image editing system based on diffusion model latent spatial frequency domain adaptive selective enhancement, according to an embodiment of the present invention. Detailed Implementation

[0017] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0018] This invention aims to address the following shortcomings in existing diffusion model latent space image editing methods: (a) structural distortion caused by uniform channel transformation; (b) fixed editing intensity failing to adapt to different image structural complexities; and (c) spectral artifacts introduced by hard-threshold frequency domain filtering. Based on this, a training-free, plug-and-play frequency domain adaptive selective enhancement image editing method and system is provided.

[0019] like Figure 1 As shown, the image editing method based on diffusion model latent spatial frequency domain adaptive selective enhancement according to an embodiment of the present invention includes the following steps: (1) Preprocessing and latent encoding; After preprocessing the image to be edited, it is encoded by a variational autoencoder (VAE) using a diffusion model to obtain the latent representation of the image in the latent space. Among them, the four channels of potential representation include two structural channels and two style channels; Specifically, for the input image to be edited Perform center cropping and scaling to the target resolution, followed by variational autoencoder. Encoding yields latent representation ; (2) Channel differentiation mapping; like Figure 2 As shown, the latent representation Four channels , , , It is divided into two functional groups: the structure channel and the style channel; the structure channel , Used only for structural complexity estimation, and does not participate in any write-back operations; Style channel , Perform frequency domain enhancement processing; (3) Adaptive editing intensity calculation; By analyzing the proportion of high-frequency energy in the structural channels, the adaptive editing intensity is automatically adjusted, resulting in more conservative editing of images with complex structures and more aggressive enhancement of images with smooth structures. Specifically, for structural channels , First, the matrix is ​​transformed from the spatial domain to the frequency domain using a two-dimensional discrete Fourier transform. Then, the zero-frequency component is moved from the upper left corner of the matrix to the center using the spectrum centering operation fftshift. The Euclidean distance from any frequency point (u, v) on the spectrum plane to the center point is calculated with the zero frequency as the center point. On the centralized spectrum, calculate Greater than or equal to the preset intermediate frequency boundary radius The proportion of high-frequency energy is obtained by taking the sum of the amplitudes at each frequency point as a percentage of the total amplitude across the entire frequency band. This value quantifies the richness of high-frequency structural information (edges, textures, details) in the structural channels, i.e., the structural complexity of the image; Based on this, the adaptive editing intensity is derived using the adaptive editing intensity adjustment formula. ; images with complex structures (high) The adaptive editing intensity obtained Editing intensity lower than the base level results in a more conservative editing effect; image structure is smoothed (lower intensity). The adaptive editing intensity obtained It offers higher editing intensity than basic editing, providing more comprehensive enhancements. Specifically, the proportion of high-frequency energy in the structural channel for: ; in, Let be the Euclidean distance from any point on the spectral plane to the zero-frequency position centered on the spectrum. The preset mid-frequency boundary radius, For two-dimensional Fourier transform, This is a potential representation of the structural channel. Indicates the corresponding structural channel , ; Furthermore, the adaptive editing intensity of this embodiment of the invention for: ; ; in, Based on editing intensity, For the clamp function, , These are the lower and upper limits of the scaling factor. The target is the proportion of high-frequency energy; like Figure 3 As shown, the mapping relationship between the proportion of high-frequency energy and editing intensity is illustrated, with the horizontal axis representing structural complexity and the vertical axis representing the scaling factor, demonstrating the inverse proportional function relationship and upper / lower pruning lines; the DUTS dataset is also displayed. Distribution, with the target value τ set to 0.28; (4) Frequency domain bandpass enhancement; The style channel is enhanced by frequency domain bandpass enhancement based on adaptive editing intensity to obtain the latent representation of the enhanced style channel. Specifically, a smooth bandpass filter is applied to the style channel. First, a spatial window function is applied to suppress boundary discontinuities, and then the intermediate frequency component is extracted by a frequency domain Gaussian bandpass filter and injected back into the original channel. The specific process of frequency domain bandpass enhancement includes: By sequentially applying the Tuck window function, Fourier transform, Gaussian bandpass filter, and two-dimensional inverse Fourier transform to the latent representation of the style channels, the enhanced latent representation of the style channels is obtained. for: ; in, As a potential representation of the style channel, Indicates the corresponding style channel , , Injecting coefficients into style, This is a two-dimensional inverse Fourier transform. Let be the frequency domain transfer function of the Gaussian bandpass filter; The frequency domain bandpass enhancement uses a Gaussian bandpass filter, constructed by the difference between two Gaussian spectra with different standard deviations: a wider Gaussian spectrum extracts low-frequency components, and a narrower Gaussian spectrum extracts even lower-frequency components; the difference between the two yields the mid-frequency bandpass response. The filter exhibits a smooth roll-off characteristic in the frequency domain, avoiding sharp frequency truncation; the default configuration is an mid-frequency boundary radius r. mid =30; Steps (2) to (4) above in this embodiment of the invention constitute three operations of the core editor: adaptive intensity calculation, frequency domain bandpass enhancement, and write-back. This editor is injected in the latent preparation stage of the diffusion model, prepared_latents, and executed before the UNet denoising loop without modifying any model weights. (5) Channel fusion; like Figure 2 As shown, this demonstrates the working mechanism of the structure channel passthrough and the style channel frequency domain enhancement. Specifically, it fuses the latent representations of the two structure channels and the two enhanced style channels to obtain the edited latent representation. for: ; (6) After the edited latent representation is denoised by UNet, it is input into the variational autoencoder for decoding to generate the output image.

[0020] Based on the above image editing methods, such as Figure 4As shown in the figure, the image editing system based on diffusion model latent space frequency domain adaptive selective enhancement of the present invention includes the following functional modules: preprocessing module (center cropping and scaling) → latent coding module (VAE encoder) → adaptive intensity module (hf_ratio estimation and scale calculation) → frequency domain editing module (windowing, FFT, Gaussian bandpass filtering, IFFT) → pipeline injection module (prepare_latents injection points) → decoding output module (VAE decoder + image output). The data flow is indicated by arrows connecting the modules. The whole system is embedded in the StableDiffusion img2img diffusion pipeline (including standard components such as UNet denoising, scheduler, and text encoder) and runs in a non-intrusive manner. The aforementioned preprocessing module is used to preprocess the image to be edited; The aforementioned latent coding module is used to encode the preprocessed image to obtain a latent representation of the image in the latent space; The aforementioned adaptive intensity module is used to automatically adjust the adaptive editing intensity by analyzing the proportion of high-frequency energy in the structural channel; The aforementioned frequency domain editing module is used to perform frequency domain bandpass enhancement on the style channel based on adaptive editing intensity, thereby obtaining the latent representation of the enhanced style channel; The pipeline injection module described above is used to fuse the latent representations of the two structural channels and the two enhanced style channels to obtain the edited latent representation; it is also used to perform UNet denoising by non-intrusive injection into the UNet denoising network; The aforementioned decoding output module is used to decode and output the latent representation after UNet denoising to generate an output image; The specific processing procedures of the above functional modules can be found in the detailed description of the image editing methods described above, and will not be repeated here.

[0021] The following detailed explanation uses specific application examples: In this embodiment of the invention, on a Stable Diffusion 1.5 VAE encoder, Fourier spectrum analysis of the four-channel latent representation revealed a clear spectral hierarchy structure among the four channels: > ≈ > (Low-frequency energy sorting); Based on this, The channel has the highest low-frequency energy concentration and is the purest channel in terms of structure; it should not be edited under any circumstances. The channel has the lowest low-frequency energy concentration and is the purest style channel, so it should always be involved in enhancement; , The channel is in an intermediate transition state. Experimental verification shows that a fixed selection... , Editing as a style channel achieves an optimal balance between color enhancement and structural fidelity.

[0022] The specific experimental comparisons are as follows: A1 (Full Channel Editing): Apply the same bandpass filter to all four channels; compared to the method in this embodiment of the invention. , The strategy resulted in a 27% degradation in LPIPS and a 15% degradation in SSIM, while color metrics showed no improvement; this directly confirms that editing the structure channel damages geometric fidelity without providing any stylistic benefits. A2 (Structure Channel Editing Only): Only for... and Applying bandpass filtering to the channel, compared to the method in the embodiments of this invention. , The strategy resulted in a 7.2% degradation in LPIPS and a 3.2% degradation in SSIM, with both color metrics falling below those of the method described in this embodiment. Together with A1, it is stated that the structure channel should not be modified, while the style channel should be edited. A3 (Randomly retain two channels): Two channels are randomly selected and retained for each image (5 trials); the method of this embodiment outperforms the random mean in 80% of LPIPS and in 76% of SSIM. A4 (Spectrum-Driven Automatic Selection): Selecting channels to retain based on low-frequency energy: The LPIPS results for A4-A (retaining the two channels with the highest energy) are almost identical to those of the method in this embodiment, but only 44% of the images are selected. , ; A4-B (retaining the two channels with the lowest energy) showed a 5.5% LPIPS degradation, and The proportion of those selected as style channels reached 100%; this further validates... Must be protected Should be edited and Interchangeable spectral hierarchy structure.

[0023] The adaptive intensity mechanism automatically calculates the scaling factor of the editing intensity during inference based on the frequency domain energy distribution of the structural channels. The more complex the structure (the higher the proportion of high-frequency energy), the smaller the scaling factor, resulting in a more conservative editing; conversely, the smoother the structure (the lower the proportion of high-frequency energy), the larger the scaling factor, resulting in a more aggressive editing. The effectiveness of this mechanism is most fully demonstrated in batch processing scenarios. In batch processing involving a large number of heterogeneous images (from product photos with solid backgrounds to cluttered outdoor scenes), disabling the adaptive mechanism requires using a globally optimal fixed intensity determined through prior experiments to achieve similar aggregation metrics. In contrast, the adaptive mechanism automatically makes parameter tuning decisions for each image during inference, requiring no additional parameters and incurring negligible computational overhead. Therefore, the value of the adaptive mechanism lies in its operability rather than statistical significance—it achieves zero-knob operation without changing the aggregation mean, which is of substantial significance for batch processing scenarios.

[0024] The default parameter configuration for frequency domain bandpass enhancement in this embodiment of the invention is: r mid = 30 (mid-frequency boundary), style injection coefficient β=0.18, basic editing intensity s =1.8, adaptive target high-frequency proportion τ=0.28, scaling range [s min , s max =[0.75, 1.8]; Ablation experiments show that bandwidth is a more dominant control factor than editing intensity. Narrowing the bandwidth from [10, 30] to [8, 24] results in a greater gain in Colorfulness (+0.0068) than increasing the editing intensity from 1.2 to 2.4 (+0.0039). Within reasonable ranges, adjusting the editing intensity (1.2–2.4) and bandwidth (narrow / wide) yields limited changes in Colorfulness and Saturation, demonstrating the method's robustness to hyperparameter selection.

[0025] In order to suppress spectral boundary artifacts in frequency domain processing, this invention preprocesses the latent representation using a cosine gradient spatial window function before performing a Fourier transform on the style channel. This window function smoothly decays to zero in the edge region, fundamentally eliminating ringing and block artifacts caused by signal boundary discontinuities. Furthermore, by combining the Gaussian spectrum smooth roll-off characteristics of the bandpass filter, the visual quality of the output image is significantly improved without increasing the computational load.

[0026] This invention is also applicable to batch image processing scenarios. The workflow is as follows: images are sampled from the input directory; each image uses a recursive seeding strategy of seed = base_seed + idx to ensure reproducibility within the batch and that the images are not completely identical in noise; in each round, the potential preparation is restored before being injected into the editor to avoid crosstalk of the editor state; the output comparison mosaic is sent to the target directory. In the batch test on the full set of 5,019 images in DUTS, the increase in average inference time of the method of this invention is negligible (only the constant overhead of frequency domain transformation, with no additional network forward propagation).

[0027] Additionally, the filtering radius in this embodiment of the invention is: r mid The value can be adjusted within the range of 20-40 to accommodate the potential space of different resolutions or VAE structures; regarding the window function type: the cosine gradient window can be replaced with other smooth gradient window functions, and the choice of window function has a mild impact on the final effect; for the adaptive parameter τ, it can be recalibrated according to the median of the dataset distribution for specific application scenarios; [s min , s max The range can be adjusted to match different editing aggressiveness requirements; moreover, the invention is designed to be pipeline-independent and plug-and-play compatible with any diffusion model that uses VAE potential space.

[0028] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. An image editing method based on diffusion model latent spatial frequency domain adaptive selective enhancement, characterized in that, Includes the following steps: S1. After preprocessing the image to be edited, it is encoded by a variational autoencoder of a diffusion model to obtain the latent representation of the image in the latent space; wherein, the four channels of the latent representation include two structure channels and two style channels; S2. By analyzing the high-frequency energy ratio of the structural channel, the adaptive editing intensity is automatically adjusted. Based on the adaptive editing intensity, the style channel is enhanced by frequency domain bandpass enhancement to obtain the potential representation of the enhanced style channel. The latent representations of the two structural channels and the two enhanced style channels are fused to obtain the edited latent representation; S3. After denoising the edited latent representation using UNet, input it into the variational autoencoder for decoding to generate the output image.

2. The image editing method according to claim 1, characterized in that, In step S1, the four channels of the latent representation are respectively , , , ; in, , For structural channels, , For style channels.

3. The image editing method according to claim 1, characterized in that, In step S1, the preprocessing includes center cropping and scaling to the target resolution.

4. The image editing method according to claim 1, characterized in that, In step S2, the latent representation of the structural channel is subjected to a two-dimensional Fourier transform and then spectral centering is performed. Then, the high-frequency energy proportion of the structural channel is calculated. for: ； Where r is the Euclidean distance from any point on the spectral plane to the zero-frequency position of the spectral center. The preset mid-frequency boundary radius, For two-dimensional Fourier transform, This is a potential representation of the structural channel. Indicates the corresponding structural channel , .

5. The image editing method according to claim 4, characterized in that, In step S2, the adaptive editing intensity for: ； in, Based on editing intensity, For the clamp function, , These are the lower and upper limits of the scaling factor. The target is the proportion of high-frequency energy.

6. The image editing method according to claim 5, characterized in that, The frequency domain bandpass enhancement process includes: The latent representation of the style channel is sequentially subjected to a Tuck window function, a Fourier transform, a Gaussian bandpass filter, and a two-dimensional inverse Fourier transform to obtain the enhanced latent representation of the style channel.

7. The image editing method according to claim 6, characterized in that, In step S2, the latent representation of the enhanced style channel for: ； in, As a potential representation of the style channel, Indicates the corresponding style channel , , Injecting coefficients into style, This is a two-dimensional inverse Fourier transform. Let be the frequency domain transfer function of the Gaussian bandpass filter.

8. The image editing method according to claim 7, characterized in that, The edited potential representation for: 。 9. An image editing system based on diffusion model latent spatial frequency domain adaptive selective enhancement, applying the image editing method as described in any one of claims 1-8, characterized in that, The image editing system includes: The preprocessing module is used to preprocess the image to be edited; The latent coding module is used to encode the preprocessed image to obtain the latent representation of the image in the latent space; The adaptive intensity module is used to automatically adjust the adaptive editing intensity by analyzing the proportion of high-frequency energy in the structural channels; The frequency domain editing module is used to perform frequency domain bandpass enhancement on the style channel based on adaptive editing intensity, so as to obtain the potential representation of the enhanced style channel. The pipeline injection module is used to fuse the latent representations of two structural channels and two enhanced style channels to obtain the edited latent representation; it is also used to perform UNet denoising by non-intrusive injection into the UNet denoising network. The decoding output module is used to decode and output the latent representation after UNet denoising to generate the output image.

Images