Image style migration method and device and storage medium

Abstract

The invention discloses an image style migration method and device and a storage medium, and relates to the technical field of computer vision. An image style migration method comprises the following steps: inputting target content information into an SD model, and embedding a style reference image into a denoising chain of the SD model; when the target content information is a text prompt, constructing a denoising modulation mechanism, and applying the denoising modulation mechanism to a denoising chain sampling process; before a denoising cycle is started, initial potential pre-adaptation is introduced, and style potential variables and content features are preliminarily aligned; through an auxiliary style text matching strategy, guiding the de-noising track to a target style domain, and generating an image after style migration; and when the target content information is a target content image, introducing a spatial perception modulation mechanism, establishing a regional mapping relation between the target content image and the style reference image, applying the regional mapping relation to a de-noising chain sampling process, and generating an image after style migration. According to the method, relatively high style migration performance is shown in text-to-image style migration or image-to-image style migration.

Description

Technical Field

[0001] This invention relates to an image style transfer method, apparatus, and storage medium, belonging to the field of computer vision technology. Background Technology

[0002] Image style transfer aims to integrate stylistic features from a reference image into a target image while preserving its core content. It has been widely applied in digital art creation, smart camera filters, and personalized design. Recent breakthroughs in diffusion models (DMs) and large-scale pre-trained text-to-image (T2I) models have revolutionized this field, enabling a text-driven style transfer paradigm that combines text guidance with reference-based style adaptation, significantly improving the flexibility and practicality of real-world applications.

[0003] Existing diffusion-based style transfer methods mainly fall into two categories. One category is training-based fine-tuning, which learns the target style by directly fine-tuning the backbone network or adding lightweight branches. While these methods perform well, they incur significant computational and storage overhead and face considerable challenges in dataset preparation—insufficient data leads to overfitting, while constructing large-scale datasets is extremely time-consuming. The other category is training-free methods, which follow two paths: one is to extract style embeddings through the CLIP image encoder and inject them through an additional cross-attention branch introduced by the adapter; however, the inherent design of CLIP causes content and style features to become entangled, resulting in insufficient style representation. The other is to obtain the intermediate denoised state of the style image through diffusion reverse engineering and achieve style transfer by performing key-value (KV) replacement in the U-Net self-attention layer. This doubles the inference latency and suffers from style degradation due to the optimization limitations of reverse engineering. Summary of the Invention

[0004] The purpose of this invention is to provide an image style transfer method, apparatus, and storage medium. By introducing a denoising modulation mechanism, initial latent pre-adaptation, and auxiliary style text matching, the invention balances the trade-off between style fidelity and content preservation in text-to-image style transfer. Furthermore, it achieves strong performance in image-to-image style transfer through a spatially perceptual modulation mechanism.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution.

[0006] On one hand, the present invention provides an image style transfer method, comprising:

[0007] Obtain style reference images and target content information, wherein the target content information includes text prompts or target content images;

[0008] The target content information is input into the SD model (diffusion model), and the style reference image is embedded into the denoising chain of the SD model;

[0009] When the target content information is a text prompt:

[0010] A denoising modulation mechanism is constructed and applied to the sampling process of the denoising chain.

[0011] Before the denoising loop begins, an initial latent prefit is introduced to initially align style latent variables with content features. The style latent variables are obtained by the VAE encoder from the style reference image, and the text features are obtained by the CLIP encoder from the text cue encoding.

[0012] By using an auxiliary style text matching strategy, the denoising trajectory is guided to the target style domain to generate a style-transferred image;

[0013] When the target content information is a target content image:

[0014] Before the denoising loop begins, a spatially aware modulation mechanism is introduced to establish a mapping relationship between the target content image and the style reference image region.

[0015] The region mapping relationship is applied to the denoising chain sampling process to generate a style-transferred image.

[0016] Optionally, the construction of the denoising modulation mechanism involves analyzing and constructing an ideal style trajectory;

[0017] Define the style latent variable as , representing the target state of the ideal style trajectory, corresponding to the final denoising state; both the content generation path and the ideal style path use shared initial noise. , As the starting point for denoising; based on the forward process of the SD model, intermediate style latent variables on the ideal style trajectory Represented as:

[0018] ;

[0019] According to the formula, the first... A time step from form Ideal style noise required Represented as:

[0020] ;

[0021] In the formula: Indicates the noise scaling factor; , These are predefined noise scheduling coefficients;

[0022] Ideal style noise middle Replace with the actual denoising latent variable of the current content generation path This forms a noise reduction modulation mechanism, represented as:

[0023] ;

[0024] Make It can be dynamically updated throughout the sampling process and adapt to trajectory changes;

[0025] Modulated actual denoising latent variable The UNet component is fed into the next time step, where it refines the latent features to reduce artifacts while maintaining semantic coherence. This synergy combines the high fidelity of analyzing style trajectories with the powerful semantic understanding capabilities of the pre-trained UNet.

[0026] Optionally, the application to the denoising chain sampling process includes: verifying the theoretical trade-off between latent variable states, intact noise, and conditional noise in terms of style controllability and content fidelity using a fixed linear mixture method, according to the formula:

[0027] ;

[0028] In the formula: Variables representing content generated from the path; yes The corresponding regulatory variables; To adjust the weights, the value is... ;

[0029] Experimental comparisons have shown that injecting ideal style into the conditional branch of noise prediction using conditional noise can maintain the integrity of the cue semantics while introducing significant style features.

[0030] Optionally, when injecting the ideal style into the conditional branch of noise prediction, a time-dependent adjustment weight is used. replace ;

[0031] Adjustment weight Adjusting the offset Parameterization is performed, and the segments are defined as follows:

[0032] ;

[0033] In the formula: , For the current time step, Maximum time step; and These are parameters; attenuation factor. satisfy Time constraints: ;

[0034] Adjustment The duration of the high-weight window is determined, thereby achieving intuitive control over the overall style intensity. The resulting dynamic modulation formula is expressed as follows:

[0035] ;

[0036] In the formula: Represents ideal style noise; This indicates conditional noise.

[0037] Optionally, the process of introducing the initial latent prefit is as follows: from the initial latent variables Departure, Execution The next iteration, represented by the following formulas, indicates the noise modulation and sampling update during the pre-adaptation phase:

[0038] ;

[0039] ;

[0040] In the formula: The iteration number is [number], and its value range is [range]. ; For the current time step, Maximum time step; Indicates conditional noise; Indicates the noise scaling factor; , These are predefined noise scheduling coefficients;

[0041] go through After the iteration, the result is As from arrive The starting point for backsampling.

[0042] A brief warm-up step effectively reduces early global bias, eliminating the need for significant model corrections in later stages. This allows the denoising process to focus on optimizing fine-grained content and high-level styles.

[0043] Optionally, the auxiliary style text matching strategy works in conjunction with a lightweight style text and a noise reduction modulation mechanism.

[0044] CLIP similarity is used to compare the style text of the style reference image with the text in the text prompt. If the CLIP similarity between the style text and any text in the text prompt exceeds a preset threshold, the two style texts with the highest matching degree corresponding to the style reference image are appended to the text prompt; otherwise, the text prompt remains unchanged.

[0045] The auxiliary style cues guide the denoising trajectory to the target style domain. This coarse-grained global guidance provides a high-probability path for the denoising process, while the denoising modulation mechanism performs fine-grained optimization to accurately generate the complex textures, colors, and compositional structures required for the target style.

[0046] Optionally, the spatially aware modulation mechanism calculates a latent spatial correspondence map for each spatial location of the target content image, specifying which region of the style reference image guides the modulation at that location;

[0047] During the denoising process, based on the mapping relationship between the target content image and the style reference image regions, each region of the target content image obtains ideal style noise from the corresponding region of the matched style reference image.

[0048] Optionally, the process of constructing the mapping relationship between the target content image and the style reference image region is as follows:

[0049] The self-attention layer of the UNet denoising network is used as a cross-domain correspondence estimator;

[0050] Extract content query features and style key features from the UNet self-attention layer;

[0051] Calculate the attention matrix between content query features and style key features;

[0052] The attention matrix is ​​optimized using a fast matching algorithm to establish a reliable mapping between local regions of the content image and the style image.

[0053] In a second aspect, the present invention provides an image style transfer apparatus, comprising:

[0054] The data acquisition module is used to: acquire style reference images and target content information, wherein the target content information includes text prompts or target content images;

[0055] The style transfer module is used to: input the target content information into the SD model and embed the style reference image into the denoising chain of the SD model;

[0056] When the target content information is a text prompt:

[0057] A denoising modulation mechanism is constructed and applied to the sampling process of the denoising chain.

[0058] Before the denoising loop begins, an initial latent prefit is introduced to initially align style latent variables with content features. The style latent variables are obtained by the VAE encoder from the style reference image, and the text features are obtained by the CLIP encoder from the text cue encoding.

[0059] By using an auxiliary style text matching strategy, the denoising trajectory is guided to the target style domain to generate a style-transferred image;

[0060] When the target content information is a target content image:

[0061] Before the denoising loop begins, a spatially aware modulation mechanism is introduced to establish a mapping relationship between the target content image and the style reference image region.

[0062] The region mapping relationship is applied to the denoising chain sampling process to generate a style-transferred image.

[0063] Thirdly, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of any of the image style transfer methods described in the first aspect.

[0064] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0065] 1. This invention is a lightweight style modulation pipeline that embeds the target style into the denoising chain of an SD model without modifying the pre-trained model itself; style transfer can utilize the model's inherent "self-editing" capability, explicitly injecting style through ideal trajectory modulation to guide the model toward the target style without extensive intervention in the generated model;

[0066] 2. In text-to-image style transfer, through the synergistic effect of denoising modulation mechanism and auxiliary style text matching strategy, this invention achieves effective alignment with target style and text semantics. In addition, compared with baseline methods, the generated images have finer texture details and brushstroke fidelity.

[0067] 3. In image-to-image style transfer, this invention provides excellent style transfer performance while maintaining the structural integrity of the target content image. Attached Figure Description

[0068] Figure 1 This is a flowchart of the image style transfer method of the present invention;

[0069] Figure 2 This is a quantitative comparison diagram of the consistency between the latent variable trajectories of the XT, FN, and CN modulation schemes of the present invention and the unmodulated baseline;

[0070] Figure 3The image shows the style transfer results of the XT, FN, and CN modulation targets of this invention.

[0071] Figure 4 The graph shows the noise prediction difference between pure content hints and style-enhanced hints in this invention, as well as the modulation control curves across time steps.

[0072] Figure 5 This is a graph showing the style transfer results of different control offsets in the control curve of this invention;

[0073] Figure 6 This is a diagram showing the difference between the denoising modulation mechanism of the present invention and the initial potential pre-adaptation state.

[0074] Figure 7 This is a schematic diagram illustrating the local matching of the target content image and the style reference image in the image-to-image style transfer process of the present invention;

[0075] Figure 8 This is a diagram showing the style transfer results of six reference styles based on text and images according to the present invention;

[0076] Figure 9 This is a qualitative comparison of the image style transfer method of the present invention and the text-based style transfer method.

[0077] Figure 10 This is a qualitative comparison of the image style transfer method of the present invention with an image-based style transfer method.

[0078] Figure 11 This is a visualization of the initial potential pre-adaptation ablation results of the present invention;

[0079] Figure 12 This is an ablation visualization of the denoising modulation mechanism and auxiliary style text matching strategy of the present invention. Detailed Implementation

[0080] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0081] Example 1:

[0082] This embodiment introduces an image style transfer method, such as... Figure 1 As shown, it includes:

[0083] Obtain style reference images and target content information, wherein the target content information includes text prompts or target content images;

[0084] The target content information is input into the SD model, and the style reference image is embedded into the denoising chain of the SD model;

[0085] When the target content information is a text prompt:

[0086] A denoising modulation mechanism is constructed and applied to the sampling process of the denoising chain.

[0087] Before the denoising loop begins, an initial latent prefit is introduced to initially align style latent variables with content features. The style latent variables are obtained by the VAE encoder from the style reference image, and the text features are obtained by the CLIP encoder from the text cue encoding.

[0088] By using an auxiliary style text matching strategy, the denoising trajectory is guided to the target style domain to generate a style-transferred image;

[0089] When the target content information is a target content image:

[0090] Before the denoising loop begins, a spatially aware modulation mechanism is introduced to establish a mapping relationship between the target content image and the style reference image region.

[0091] The region mapping relationship is applied to the denoising chain sampling process to generate a style-transferred image.

[0092] The proposed denoising modulation mechanism is constructed by analyzing and building an ideal style trajectory;

[0093] Define the style latent variable as , representing the target state of the ideal style trajectory, corresponding to the final denoising state; both the content generation path and the ideal style path use shared initial noise. , As the starting point for denoising; based on the forward process of the SD model, intermediate style latent variables on the ideal style trajectory Represented as:

[0094] ;

[0095] According to the formula, the first... A time step from form Ideal style noise required Represented as:

[0096] ;

[0097] In the formula: Indicates the noise scaling factor; , These are predefined noise scheduling coefficients;

[0098] Will Replace with the actual denoising latent variable of the current content generation path This forms a noise reduction modulation mechanism, represented as:

[0099] ;

[0100] Make It can be dynamically updated throughout the sampling process and adapt to trajectory changes;

[0101] Modulated actual denoising latent variable The UNet component is fed into the next time step, where it refines the latent features to reduce artifacts while maintaining semantic coherence. This synergy combines the high fidelity of analyzing style trajectories with the powerful semantic understanding capabilities of the pre-trained UNet.

[0102] The application to the denoising chain sampling process includes: using a fixed linear mixture method to verify the theoretical trade-off between latent variable states, intact noise, and conditional noise in terms of style controllability and content fidelity, according to the formula:

[0103] ;

[0104] In the formula: Variables representing content generated from the path; yes The corresponding regulatory variables; To adjust the weights, the value is... ;

[0105] By setting Experiments were conducted to compare the effects of three control methods: latent variable state (XT), intact noise (FN), and conditional noise (CN). The results are as follows: Figure 2 and Figure 3 As shown, XT modulation leads to severe content collapse, while FN modulation produces negligible style changes. In contrast, CN modulation achieves the optimal balance: introducing significant style features while maintaining the integrity of the cue semantics. Therefore, confirmation conditional noise constitutes the most stable and effective intervention target in denoising modulation.

[0106] When injecting the ideal style into the conditional branch of noise prediction, time-dependent adjustment weights are used. replace ;

[0107] For design This study analyzed the difference in predicted noise between pure content-based cues and style-enhanced cues. By calculating the spatial mean of the predicted noise to measure the global style influence, the mechanism by which the diffusion model spontaneously integrates style elements under textual guidance was revealed. Figure 4As shown, this difference exhibits a sharp peak in the early time steps, then gradually decays. Therefore, the regulation curve... The design is tailored to match this sensitivity to experience: maximizing style injection during the phase when the model is most perceptive, and gradually diminishing it during the later phase when content refinement is the primary focus.

[0108] Adjustment weight Adjusting the offset Parameterization is performed, and the segments are defined as follows:

[0109] ;

[0110] In the formula: , For the current time step, Maximum time step; Attenuation factor satisfy Time constraints: ;

[0111] like Figure 5 As shown, adjust The duration of the high-weight window is determined, thereby achieving intuitive control over the overall style intensity. The resulting dynamic modulation formula is expressed as follows:

[0112] ;

[0113] In the formula: Represents ideal style noise; This indicates conditional noise.

[0114] The process of introducing the initial latent prefit is as follows: from the initial latent variables... Departure, Execution The next iteration, represented by the following formulas, indicates the noise modulation and sampling update during the pre-adaptation phase:

[0115] ;

[0116] ;

[0117] In the formula: The iteration number is [number], and its value range is [range]. ; For the current time step, Maximum time step; Indicates conditional noise; Indicates the noise scaling factor; , These are predefined noise scheduling coefficients;

[0118] go through After the iteration, the result is As from arrive The starting point for backsampling.

[0119] A brief warm-up step effectively reduces early global bias, eliminating the need for significant model corrections in later stages. This allows the denoising process to focus on optimizing fine-grained content and high-level styles.

[0120] like Figure 6 As shown, without pre-adaptation ( Figure 6 (Top-middle figure) The overall values ​​of the difference plot are low, indicating a global shift in the latent representation in the early time steps; while after pre-adaptation ( Figure 6 (See the lower part of the figure). The differences in the initial latent variables mainly appear in the foreground region, which is related to the high-level style changes, while the background region retains the established low-level style characteristics.

[0121] The auxiliary style text matching strategy works in conjunction with a lightweight style text and a noise reduction modulation mechanism.

[0122] CLIP similarity is used to compare the style text of the style reference image with the text in the text prompt. If the CLIP similarity between the style text and any text in the text prompt exceeds a preset threshold, the two style texts with the highest matching degree corresponding to the style reference image are appended to the text prompt; otherwise, the text prompt remains unchanged.

[0123] The auxiliary style cues guide the denoising trajectory to the target style domain. This coarse-grained global guidance provides a high-probability path for the denoising process, while the denoising modulation mechanism performs fine-grained optimization to accurately generate the complex textures, colors, and compositional structures required for the target style.

[0124] The spatially aware modulation mechanism calculates a latent spatial correspondence map for each spatial location of the target content image, specifying which region of the style reference image guides the modulation at that location;

[0125] During the denoising process, based on the mapping relationship between the target content image and the style reference image regions, each region of the target content image obtains ideal style noise from the corresponding region of the matched style reference image.

[0126] The process of constructing the mapping relationship between the target content image and the style reference image region is as follows:

[0127] The self-attention layer of the UNet denoising network is used as a cross-domain correspondence estimator;

[0128] Extract content query features and style key features from the UNet self-attention layer;

[0129] Calculate the attention matrix between content query features and style key features;

[0130] like Figure 7 As shown, the attention matrix is ​​optimized by a fast matching algorithm to establish a reliable mapping between local regions of the content image and the style image.

[0131] Example 2:

[0132] This embodiment describes the experimental process of an image style transfer method:

[0133] I. Experimental Setup

[0134] Experiments were conducted on text / image-driven style transfer tasks against public and custom benchmarks. For text-driven tasks, cross-validation was performed using StyleBench (73 categories, 5–7 variations per category, totaling 490; e.g., painting, 3D rendering) and 40 style benchmarks from StyleID. For image-driven tasks, StyleBenchmini (derived from StyleBench) was constructed, containing 40 content images and one representative style image per category (totaling 73). All evaluations employed qualitative visual comparisons and quantitative metrics.

[0135] Specific baselines for the task / backbone network were adopted: text-driven style transfer methods included DEADiff and StyleShot style transfer methods based on Stable Diffusion version 1.5, and CSGO, InstantStyle, and Style-Aligned style transfer methods based on Stable Diffusion XL; image-driven style transfer methods included InstantStyle, StyleShot, and StyleID.

[0136] II. Experimental Results

[0137] Qualitative experimental results:

[0138] Figure 8 The results of the inventive method (DemoStyle) based on SD1.5 or SDXL are presented in six different styles, where the stylized images in each style are generated from the same text prompts. For a fair comparison, results from other text-based style transfer methods are also included. Figure 9As shown, all methods are evaluated on three style reference images, each paired with two different text cues. DemoStyle effectively captures the full stylistic features of the reference images, from basic visual elements (e.g., color palette, texture) to complex components (e.g., layout structure, lighting effects), thereby generating stylized images that are highly consistent with the text semantics.

[0139] for Figure 9 For the first two rows of style references, DemoStyle demonstrates a strong ability to mimic the color scheme and layout of the reference images, with the generated subjects being closest to the references in terms of texture consistency. For specific style backgrounds unrelated to the text semantics, DemoStyle preserves them to enhance style inheritance; conversely, when original style elements conflict with the text content, these conflicting elements are adjusted to conform to the text's main body during denoising, while preserving the target style. This confirms the previous conclusion: through the DemoStyle pipeline, the noise prediction model selectively preserves irrelevant style elements and edits conflicting elements. For the middle two rows of Fauvist painting style, DemoStyle almost perfectly inherits the color features of the reference images. While StyleShot also captures the core color palette, DemoStyle is superior in terms of brushstroke texture consistency with the reference images. For the bottom two rows of icon styles, DemoStyle shows a clear advantage, achieving a balance between accurate text alignment and natural style integration.

[0140] Content images inherently fix semantics and layout, limiting the ability of diffusion models to adjust style elements based on textual cues. However, DemoStyle still performs well in image-driven style transfer tasks thanks to local matching mechanisms. Figure 10 As shown, DemoStyle can transfer various styles, including 3D rendering, pixel art, animation, and painting, to diverse content images such as people, animals, and scenes. Compared to StyleID, which uses inversion and attention replacement, DemoStyle outperforms in text responsiveness and style alignment. While adapter-based methods like InstantStyle and StyleShot achieve smooth transfer of global color and layout, DemoStyle excels in preserving high-level style features. This confirms that DemoStyle delivers superior style transfer performance while maintaining the structural integrity of content images.

[0141] Quantitative experimental results:

[0142] User preference studies were conducted to evaluate text / style alignment (text-based) and content-preserving style transfer (image-based) tasks. Results are summarized in Tables 1 and 2, respectively. For completeness, CLIP scores are also reported; these results are for reference only and are listed in Tables 1 and 2, respectively.

[0143] Table 1. Quantitative comparison with state-of-the-art text-driven style transfer methods.

[0144]

[0145] Table 2. Quantitative comparison with state-of-the-art image-based style transfer methods.

[0146]

[0147] For text-driven style transfer, DemoStyle significantly outperforms in text / style alignment, demonstrating its strong stylization capabilities across multiple styles and high sensitivity to text. For image-driven tasks, DemoStyle ranks second in style alignment (only behind StyleShot), indicating that its local matching mechanism effectively alleviates the challenge of "hard transfer" with fixed content. Further optimization is still possible in balancing content and style.

[0148] The DemoStyle text-driven style adjustment pipeline has three adjustable hyperparameters: auxiliary text activation, latent pre-adaptation enable, and adjustment curve parameters. By leveraging the diverse style categories in StyleBench, a hyperparameter library was constructed: for unseen style images, CLIP can quickly match the most similar style to retrieve near-optimal hyperparameters, requiring only minor fine-tuning for further performance improvements. As shown in Table 3, this library-based matching strategy maintains competitive CLIP scores and excellent visual quality for style images provided by StyleID without manual adjustments.

[0149] Table 3. Quantitative assessment of text-to-image style transfer on StyleID images.

[0150]

[0151] Ablation experiment:

[0152] Modulation curve parameters It controls how much style is injected during the denoising process. Ablation experiments are conducted on the proposed initial potential prefit to demonstrate its ability to tame adjustments for multiple styles. A tricky problem at the time. Figure 11 Four typical failure scenarios for direct modulation curve migration are presented. Two methods were tested for each style. system:

[0153] keep In the absence of initial potential pre-adaptation ( Figure 11 In the case of the middle left column, the intensity is insufficient: only local style patches appear. With the initial latent pre-adaptation, the latent content is already style-aligned before the first denoising step, thus maintaining the same conservatism. It is now possible to transfer the complete style while retaining the content.

[0154] radical Push the curve higher ( Figure 11 The middle right column will suddenly be injected with a style signal that the noise model cannot correct; this will cause obvious artifacts. Figure 11 (Line 1) or partially blurred ( Figure 11 (Line 2). When initial latent prefitting is enabled, the latent values ​​are pre-adjusted, thus conservative. That's it; the aggressive mode is no longer needed; quality has consistently improved.

[0155] Therefore, the initial potential prefit transforms an unstable and style-specific tuning problem into a single, reliable, and conservative setting.

[0156] pass Figure 12 The visual evaluation assessed the gain effect of auxiliary style cues in denoising modulation: "Baseline" used only the original cues; "Denoising Modulation" added modulation to the baseline; and "Auxiliary Cue" further introduced matching auxiliary style cues. Compared to the baseline, denoising modulation successfully transferred major style features (such as color layout) while maintaining text alignment, but suffered from style texture distortion (such as dynamic textures in 3D rendering; naturalism bias in painting). After adding auxiliary style cues, both texture fidelity and overall style quality were significantly improved. Figure 12 (Middle and last column)

[0157] Example 3:

[0158] Based on the same inventive concept as Embodiment 1, this embodiment introduces an image style transfer apparatus, comprising:

[0159] The data acquisition module is used to: acquire style reference images and target content information, wherein the target content information includes text prompts or target content images;

[0160] The style transfer module is used to: input the target content information into the SD model and embed the style reference image into the denoising chain of the SD model;

[0161] When the target content information is a text prompt:

[0162] A denoising modulation mechanism is constructed and applied to the sampling process of the denoising chain.

[0163] Before the denoising loop begins, an initial latent prefit is introduced to initially align style latent variables with content features. The style latent variables are obtained by the VAE encoder from the style reference image, and the text features are obtained by the CLIP encoder from the text cue encoding.

[0164] By using an auxiliary style text matching strategy, the denoising trajectory is guided to the target style domain to generate a style-transferred image;

[0165] When the target content information is a target content image:

[0166] Before the denoising loop begins, a spatially aware modulation mechanism is introduced to establish a mapping relationship between the target content image and the style reference image region.

[0167] The region mapping relationship is applied to the denoising chain sampling process to generate a style-transferred image.

[0168] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0169] Example 4:

[0170] Based on the same inventive concept as other embodiments, this embodiment describes a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the image style transfer method as described in any of the embodiments.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image style transfer method, characterized in that, The method comprises the following steps: obtaining a style reference image and target content information, wherein the target content information comprises a text prompt or a target content image; inputting the target content information into an SD model and embedding the style reference image into a denoising chain of the SD model; when the target content information is a text prompt: constructing a denoising modulation mechanism and applying the denoising modulation mechanism to a sampling process of the denoising chain; before starting a denoising cycle, introducing an initial latent pre-adaptation to preliminarily align a style latent variable and a content feature, wherein the style latent variable is obtained by extracting the style reference image by a VAE encoder, and the content feature is obtained by encoding the text prompt by a CLIP encoder; guiding a denoising trajectory to a target style domain by an auxiliary style text matching strategy to generate a style-transferred image; when the target content information is a target content image: before starting the denoising cycle, introducing a spatial perception modulation mechanism to establish a region mapping relationship between the target content image and the style reference image; applying the region mapping relationship to the sampling process of the denoising chain to generate the style-transferred image.

2. The image style transfer method of claim 1, wherein, The denoising modulation mechanism is constructed by analyzing an ideal style trajectory; The style latent variable is defined as , a target state representing an ideal style trajectory, corresponding to a final denoised state; both the content generation path and the ideal style path use a shared initial noise , as a starting point for denoising; intermediate style latent variables on the ideal style trajectory are represented as: ​ ； According to the formula, at the first time step, the formation of the desired style noise is represented as: ； In the formula: denotes a noise scaling factor; , is a predefined noise scheduling coefficient; replacing the ideal style noise in the actual denoising latent variable of the current content generation path , forming a denoising modulation mechanism, denoted as: 。 3. The image style transfer method of claim 1, wherein, the application to the sampling process of the denoising chain comprises: verifying a theoretical trade-off between style controllability and content fidelity of a latent variable state, complete noise and conditional noise by using a fixed linear mixing method, and determining a condition branch for injecting ideal style noise into noise prediction according to a formula: ； In the formula: denotes a variable from the content generation path; is a corresponding control variable; is a control weight, with a value of ; the auxiliary style text matching strategy is to provide a lightweight style text to work with the denoising modulation mechanism; 4. The image style transfer method of claim 3, wherein, The time-dependent regulating weight is adopted when the ideal style is injected into the condition branch of noise prediction substituted ; Regulation weight By regulation offset Parametrization, segment definition: ； wherein: , is the current time step, is the maximum time step; and are parameters, respectively. Attenuation factor Satisfies Constraints at time: ; adjustment The duration of the high weight window is determined, and the final dynamic modulation formula is expressed as: ； In the formulae: represents the ideal style noise; represents the conditional noise.

5. The image style transfer method of claim 1, wherein, The process of introducing the initial latent preadaptation is to perform a first micro iteration, where the noise modulation and sample update in the preadaptation phase are represented by the following equations, respectively: ​ ； ； In the formula: is the iteration number, and the value range is ; is the current time step, is the maximum time step; represents the conditional noise; represents the noise scaling factor; , is a predefined noise scheduling coefficient; After the next iteration, the resulting is used as the starting point for the reverse sampling from to .

6. The image style transfer method of claim 1, wherein, a CLIP similarity is used to compare a style text of the style reference image with texts in the text prompt, and when the CLIP similarity between the style text and any text in the text prompt exceeds a preset threshold, the style text corresponding to the style reference image with the highest matching degree is added to the text prompt; otherwise, the text prompt remains unchanged. The spatial perception modulation mechanism calculates a latent space corresponding relationship map for each spatial position of the target content image, and specifies which region of the style reference image guides the modulation of the position; 7. The image style transfer method of claim 1, wherein, in the denoising process, according to the region mapping relationship between the target content image and the style reference image, each region of the target content image obtains ideal style noise from the corresponding region of the matched style reference image. The construction process of the region mapping relationship between the target content image and the style reference image comprises the following steps:

8. The image style transfer method of claim 1, wherein, using a self-attention layer of a UNet denoising network as a cross-domain corresponding relationship estimator; extracting a content query feature and a style key feature from the self-attention layer of the UNet; calculating an attention matrix between the content query feature and the style key feature; optimizing the attention matrix by a fast matching algorithm to establish a reliable mapping between local regions of the content image and the style image. The method comprises the following steps:

9. An image style transfer apparatus, characterized by, a data acquisition module is configured to obtain a style reference image and target content information, wherein the target content information comprises a text prompt or a target content image; a style transfer module is configured to input the target content information into an SD model and embed the style reference image into a denoising chain of the SD model; when the target content information is a text prompt: constructing a denoising modulation mechanism and applying the denoising modulation mechanism to a sampling process of the denoising chain; ​ Before the start of the denoising cycle, an initial latent pre-adaptation is introduced for the preliminary alignment of the style latent variable extracted by the VAE encoder on the style reference image and the content feature extracted by the CLIP encoder on the text prompt; Through an auxiliary style text matching strategy, the denoising trajectory is guided to the target style domain to generate a style-transferred image; When the target content information is a target content image: Before the start of the denoising cycle, a spatial perception modulation mechanism is introduced to establish a regional mapping relationship between the target content image and the style reference image; The regional mapping relationship is applied to the denoising chain sampling process to generate a style-transferred image.

10. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the image style transfer method of any one of claims 1-8.

Images