A Customized Conceptual Replacement Method Based on Stable Diffusion and Fractional Distillation

Abstract

This invention proposes an efficient, customized concept replacement method, InstantSwap, which handles shape differences between source and target concepts while maintaining foreground-background consistency without training. The method includes automatically generating bounding boxes to locate the source concept, applying a background gradient mask to protect the background region, enhancing foreground consistency through semantically augmented concept representations, and performing concept replacement. Furthermore, a skip gradient update (SSGU) strategy is introduced, calculating gradients only at key time steps to reduce forward propagation and improve efficiency. Through extensive qualitative and quantitative evaluations, as well as comprehensive ablation studies, this invention demonstrates the effectiveness and superiority of its method, showcasing its versatility and broad applicability in relevant tasks. The method is particularly suitable for applications requiring precise concept replacement, such as selfie enhancement, photo blogging, and comic creation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image editing, and particularly relates to a customized concept replacement method based on stable diffusion and fractional distillation. BACKGROUND

[0002] Image editing is a fundamental and popular topic in the field of computer vision. Previous research based on generative adversarial networks (GAN) only focuses on specific object domains, which limits its application. With the emergence of diffusion models, image editing can now modify various objects through prompts. These methods can be mainly divided into five categories: instruction-based methods, hybrid-based methods, attention-based methods, inversion-based methods, and fractional distillation-based methods. Instruction-based methods (instructpix2pix, instructdiffusion, smart edit, magic brush) usually require instruction editing datasets to train diffusion models. Hybrid-based methods combine source and target prompts to guide the editing process, while attention-based methods inject attention features from source images. Both methods have lower editing costs, but have poor background preservation and prompt alignment. Inversion-based methods aim to reverse the fixed trajectory generated by forward propagation to reproduce the source image. These methods can be used as an additional training stage to enhance the background consistency of attention-based methods. Finally, fractional distillation-based methods draw on the optimization process of and use fractional distillation-based losses to optimize source images for editing. These methods are more flexible than previous methods, but still face challenges in background preservation.

[0003] Customized Concept Replacement (CCS), a subtask of Text-to-Image (T2I) generation, aims to replace the concept in the source image with a highly customized new concept. Combined with diffusion models, recent customized concept replacement methods have shown wide applicability in selfie beautification, photo blog creation, and comic creation.

[0004] Early customized concept replacement work mainly relies on copy-paste techniques, which are rough and unreliable. By combining powerful customization techniques with image editing methods, a series of research results have been proposed. Despite significant success, these methods still face the problems of inconsistency and inefficiency. For example, Figure 1As shown: (1) Inconsistency: Attention-based methods such as PhotoSwap and P2P can maintain background consistency well, but perform poorly in handling shape differences between source and target concepts, resulting in inconsistent foreground. Score distillation-based methods such as SDS, DDS and CDS fail to accurately generate foreground concepts and significantly change the background, resulting in double inconsistency of foreground and background. (2) Inefficiency: Attention-based methods require an inefficient training phase on the source image to maintain background consistency. Although score distillation-based methods do not require training, they still require redundant forward pass calculations at each time step, resulting in low inference efficiency.

[0005] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The main purpose of the present application is to overcome the defects existing in the background art, and provide a customized concept replacement method based on stable diffusion and fractional distillation.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] A customized concept replacement method based on stable diffusion and fractional distillation, comprising the following steps:

[0009] S1. Automatic bounding box generation: feature extraction is performed on the source image to generate an image feature map; the cross-attention mechanism is used to determine the position of the source concept in the image in combination with the source image features and the text prompt of the source concept; and a bounding box bbox indicating the source concept is automatically generated according to the determined position;

[0010] S2. Background gradient mask BGM: using the generated bounding box bbox, the foreground and background regions are defined; and a gradient mask is applied to the background region outside the foreground region to prevent modification of the background region during the concept replacement process;

[0011] S3. Semantic enhanced concept representation: in the foreground region defined by the bounding box bbox, semantic embeddings obtained by combining the foreground region features and the text prompt are injected into the cross-attention map of the source concept and the target concept, and the representations of the source concept and the target concept are enhanced using semantic information to improve the consistency of the foreground region in the concept replacement;

[0012] S4. Concept replacement execution: using the source concept position determined by the bounding box bbox generated in step S1, and the enhanced background and foreground consistency information in steps S2 and S3, the concept replacement process is executed.

[0013] Furthermore, in step S1, the U-Net network is used to extract features from the source image.

[0014] Furthermore, step S1 specifically includes:

[0015] The U-Net network is used to perform a forward pass on the source image to obtain image features;

[0016] By combining image features and textual cues of source concepts, cross-attention maps and self-attention maps are calculated;

[0017] The cross-attention map is modified to enhance the activation of the target region through element-wise exponentiation.

[0018] Apply thresholding to obtain a coarse-grained mask;

[0019] The coarse-grained mask is converted into a precise bounding box to indicate the location of the source concept in the source image.

[0020] Further, in step S1, noise predictions are generated for the source image and the target image based on their respective text prompts; in step S2, background gradient masking (BGM) is performed using the generated noise predictions and the automatically acquired bounding boxes (bboxes) of the source concepts to maintain background consistency.

[0021] Furthermore, step S2 specifically includes:

[0022] The gradient is calculated using the latent features of the source and target images to reflect the differences between the latent features. Preferably, when calculating the gradient, a simplified model is applied to ignore the Jacobian term of the U-Net, simplifying the gradient calculation process to a constant multiplied by the difference in latent features.

[0023] The calculated gradient is combined with the bounding box (bbox) of the source image, and a background gradient mask (BGM) is generated through element-wise multiplication.

[0024] Background-related gradients are masked using background music (BGM) to ensure that gradient updates only occur within the foreground region.

[0025] Furthermore, step S3 specifically includes:

[0026] Adjust the size of the previously obtained bounding box (bbox) to fit the size of the source image features to form the feature bounding box (feature bbox);

[0027] Use feature bounding boxes to crop source image features to extract region image features related to the source concept;

[0028] Calculate the query vector using region image features and textual hints of source concepts;

[0029] Calculate the key vector and value vector based on the semantic embedding of the query vector and the source concept;

[0030] By using a cross-attention mechanism, the semantic information of the source concept is injected into the region image features to obtain the region concept representation;

[0031] By mapping the region concept representation back to the original feature map, a semantically enhanced representation of the entire source image is obtained. Using the semantically enhanced concept representation, accurate and seamless concept replacement is achieved in the foreground region to enhance the consistency of the foreground.

[0032] Furthermore, in step S4, skip-step gradient updates are performed: during the concept replacement process, a given time step is selected to perform forward propagation to calculate the gradient; in time steps where direct gradient calculation is not performed, the previously calculated gradient is reused for parameter updates; by skipping gradient calculations in non-critical time steps, the total number of forward propagations is reduced, thereby improving the efficiency of the replacement process.

[0033] Furthermore, step S4 specifically includes:

[0034] Define a computational model for the entire concept replacement process, using the source image, intermediate target image, and time step as input;

[0035] Perform a forward pass at a given time step and compute gradient updates;

[0036] The step period is determined to skip gradient calculations in adjacent time steps. The step period is controlled by a preset SSGU factor.

[0037] For time steps where direct gradient calculation is not performed, the anchor gradient calculated at the most recent time step is reused for parameter updates.

[0038] A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the customized concept substitution method based on stable diffusion and fractional distillation.

[0039] A computer program product includes a computer program, characterized in that the computer program, when executed by a processor, implements the customized concept substitution method based on stable diffusion and fractional distillation.

[0040] The present invention has the following beneficial effects:

[0041] This invention provides a customized concept replacement method based on stable diffusion and fractional distillation. It proposes an innovative, training-free customized concept replacement framework called InstantSwap, which effectively performs efficient concept replacement while maintaining foreground and background consistency. This framework generates precisely localized source concepts through automatic bounding box generation and utilizes a Background Gradient Masking (BGM) strategy to prevent unnecessary modifications to the background, ensuring background stability. Simultaneously, it enhances the semantic information of the foreground region through Semantic Enhanced Concept Representation (SECR), improving the consistency and accuracy of foreground replacement. Furthermore, this invention introduces a Stepped Skip Gradient Update (SSGU) strategy, which significantly reduces the total number of forward passes by performing forward pass gradient calculations at specific time steps and reusing previously calculated gradients at other time steps, thereby improving overall inference efficiency. This method not only reduces redundant computation but also maintains visual coherence and semantic consistency of the image during concept replacement, making customized concept replacement widely applicable in various application scenarios such as selfie enhancement, photo blog creation, and comic creation. Through extensive qualitative and quantitative evaluations, as well as comprehensive ablation studies, this invention demonstrates the effectiveness and superiority of its method, showcasing its versatility and broad applicability in relevant tasks.

[0042] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0043] Figure 1 A comparison diagram is provided to illustrate how the method of this embodiment achieves better exchange consistency than existing methods.

[0044] Figure 2 This is a general flowchart of a customized conceptual substitution method based on stable diffusion and fractional distillation, according to an embodiment of the present invention.

[0045] Figure 3 This is a comparison diagram between the SSGU of this invention and previous methods.

[0046] Figure 4 This is a diagram showing the qualitative comparison results of an embodiment of the present invention.

[0047] Figure 5 The qualitative results of the ablation study in this embodiment of the invention are shown on the left, which shows the effects of applying BGM and SECR, and on the right, which shows the effects of using different bbox generation methods.

[0048] Figure 6 Different SSGU factors in embodiments of the present invention λ The qualitative result. Too high. λ This will lead to a decrease in the consistency of the outlook. Detailed Implementation

[0049] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0050] Concept swapping is a subtask of general image editing, primarily replacing source concepts in an image with user-specified target concepts. This task was first proposed by PbE, which uses a CLIP encoder to extract features of the target concept and injects them into a UNet via a cross-attention layer. Subsequently, parallel works have extended concept swapping to the customization domain. They combine attention-based editing methods with adjustment-based customization methods to achieve customized concept swapping. Building on Photoswap, SwapAnything further leverages external modules to obtain masks to specify the location of objects in the source image. This invention improves upon existing methods in several important aspects. First, it uses bounding boxes instead of masks as spatial indicators of the source concept, providing greater flexibility for shape changes during concept swapping. Second, it uses bounding boxes with gradient masks to prevent background changes, ensuring background consistency. Third, it utilizes semantic information to enhance concept representations to maintain foreground consistency. Finally, instead of performing forward passes at every time step, it performs them only at specific intervals to improve efficiency.

[0051] This invention provides a customized concept substitution method based on stable diffusion and fractional distillation, comprising the following steps:

[0052] Step S1. Automatic bounding box generation: Extract features from the source image to generate an image feature map; use the cross attention mechanism, combined with the source image features and textual hints of the source concept, to determine the position of the source concept in the image; based on the determined position, automatically generate a bounding box (bbox) indicating the source concept.

[0053] In a preferred embodiment, step S1 uses a U-Net network to extract features from the source image. Further, step S1 specifically includes: performing a forward pass on the source image using the U-Net network to obtain image features; combining the image features and textual cues of the source concept to calculate a cross-attention map and a self-attention map; modifying the cross-attention map by enhancing the activation of the target region through element-wise exponentiation; applying a thresholding operation to obtain a coarse-grained mask; and converting the coarse-grained mask into a precise bounding box to indicate the location of the source concept in the source image.

[0054] Step S2. Background Gradient Masking (BGM): Using the generated bounding box (bbox), define the foreground and background regions; apply gradient masking to the background region outside the foreground region to prevent modification of the background region during concept replacement.

[0055] In a preferred embodiment, step S2 specifically includes: using the latent features of the source image and the target image to calculate the gradient to reflect the difference between the latent features; preferably, when calculating the gradient, a simplified model is applied to ignore the Jacobian term of the U-Net, simplifying the gradient calculation process to the form of a constant multiplied by the difference in latent features; the calculated gradient is combined with the bounding box (bbox) of the source image, and a background gradient mask (BGM) is generated through element-wise multiplication; the BGM is applied to mask the gradient related to the background, ensuring that gradient updates only occur in the foreground region.

[0056] In a preferred embodiment, in step S1, noise predictions are generated for the source image and the target image based on their respective text prompts; in step S2, background gradient masking (BGM) is performed using the generated noise predictions and the automatically acquired bounding boxes (bboxes) of the source concepts to maintain background consistency.

[0057] Step S3. Semantic Enhanced Concept Representation (SECR): Within the foreground region defined by the bounding box (bbox), the semantic embedding obtained by combining foreground region features and textual cues is injected into the cross-attention map of the source and target concepts. This enhances the representation of the source and target concepts using semantic information, thereby improving the consistency of the foreground region in concept substitution.

[0058] In a preferred embodiment, step S3 specifically includes: adjusting the size of the previously obtained bounding box (bbox) to fit the size of the source image features, forming a feature bounding box (feature bbox); cropping the source image features using the feature bounding box to extract region image features related to the source concept; calculating a query vector using the region image features and textual hints of the source concept; calculating a key vector and a value vector based on the query vector and the semantic embedding of the source concept; injecting the semantic information of the source concept into the region image features through a cross-attention mechanism to obtain a region concept representation; mapping the region concept representation back to the original feature map to obtain a semantically enhanced representation of the entire source image; and using the semantically enhanced concept representation to achieve accurate and seamless concept replacement within the foreground region to enhance foreground consistency.

[0059] Step S4. Concept replacement execution: Using the source concept location determined by the bounding box (bbox) generated in step S1, and the enhanced background and foreground consistency information in steps S2 and S3, perform the concept replacement process.

[0060] In a preferred embodiment, step S4 involves performing a step-skipping gradient update (SSGU): during the concept replacement process, a given time step is selected to perform forward propagation to calculate the gradient; in time steps where direct gradient calculation is not performed, the previously calculated gradient is reused to update the parameters; by skipping gradient calculations in non-critical time steps, the total number of forward propagations is reduced, thereby improving the efficiency of the replacement process.

[0061] Furthermore, the skip gradient update method in step S4 specifically includes: defining a computational model for the entire concept substitution process, using the source image, intermediate target image, and time step as input; performing forward propagation at a given time step to compute gradient updates; determining a skip period to skip gradient computation at adjacent time steps, the skip period being controlled by a preset SSGU factor; and reusing the anchor gradient computed at the most recent time step for parameter updates at time steps where direct gradient computation has not been performed.

[0062] This invention provides a customized concept replacement method based on stable diffusion and fractional distillation, proposing InstantSwap, a training-free framework that efficiently performs customized concept replacements across shape differences while maintaining foreground and background consistency. Specifically, this invention extracts bounding boxes (bboxes) indicating the location of source concepts from the enhanced cross-attention map of the source image. Using these bboxes, a background gradient masking (BGM) strategy is implemented to prevent modification of parts outside the bbox, thus ensuring background consistency. Furthermore, to improve foreground consistency, semantic information is used to highlight the cross-attention maps of the source and target concepts within the bboxes, respectively. This strategy results in a semantically enhanced concept representation (SECR), contributing to accurate foreground replacement. Further, a step-skipped gradient update (SSGU) strategy is introduced, performing forward passes only at specific time steps to compute gradients. For time steps without direct gradient computation, previously obtained gradients are reused for updates. This strategy reduces the total number of forward passes, improving the efficiency of the method.

[0063] Extensive qualitative and quantitative comparisons demonstrate the effectiveness and superiority of the method of this invention. Comprehensive ablation studies validate the effectiveness of each component in the method of this invention. Furthermore, the extension of InstantSwap to relevant tasks further demonstrates its efficacy and versatility.

[0064] The following further describes specific embodiments, algorithm examples, and experimental verifications of the present invention.

[0065] Figure 2The overall flow of a customized conceptual substitution method based on stable diffusion and fractional distillation is shown.

[0066] This invention proposes a novel training-free Customized Concept Exchange (CCS) framework capable of efficient concept exchange across significant shape differences. A Background Gradient Masking (BGM) strategy and Semantic Enhanced Concept Representation (SECR) are designed to improve the consistency between the background and foreground, respectively. Furthermore, a Stepped Skip Gradient Update (SSGU) strategy is employed to reduce redundant computation and improve efficiency.

[0067] In some embodiments, the bounding box (bbox) of the source concept is first automatically acquired. The acquired bbox is input into the SECR in the source and target branches to enhance foreground exchange consistency. In some embodiments, the source and target branches generate noise predictions for the source and target images based on their respective cues. The predicted noise, along with the bbox, is used in the background music (BGM) to maintain background consistency.

[0068] 1. Automatic bounding box generation

[0069] First, the bounding box is automatically obtained to indicate the concept O. s Location in the source image.

[0070] Given source image x s Source hint P s Using U-Net∈ φ (·) Perform the forward pass and obtain the cross-attention graph A as follows: c And self-attention map A s :

[0071]

[0072] Where Q is the query vector projected from the image features, and d represents the output dimension of the key and query features.

[0073] K is the key vector, and V is the value vector. For the cross-attention graph A... c K and V are projected from the text embedding τ(P). For the self-attention map A s V and V are projections from image features. Applying a threshold directly to SA^c$ produces a coarse-grained mask that cannot accurately reflect O. s The position of A. c The following modifications were made:

[0074]

[0075] According to Equation 1, A c All values ​​in A are between 0 and 1. Therefore, for A c Element-wise exponentiation by α can weaken activation in non-target regions. Furthermore, As It contains rich structural information. This information can effectively help... To better activate the target region, a threshold β is finally applied. To obtain the mask, the mask is then converted to a bounding box (bbox) based on the minimum and maximum coordinates of all foreground points within the mask. s This strategy enables the achievement of bbo without adding any modules. x B s A relatively loose constraint is intentionally set on the mask to obtain a bbox that fully covers the source concept.

[0076] 2. Background gradient masking

[0077] Using the bounding box (bbox) of an object, a background gradient masking (BGM) method is proposed to ensure that concept exchange is confined to the foreground region. Given the latent features of the source image... and the latent features z of the target image, where z is initialized as And continuously optimize to obtain the final target image x t .

[0078] Based on DDS, the gradient of z is first obtained:

[0079]

[0080] As described in SDS, the intermediate term is the U-Net Jacobian term, which can be omitted, and It is a constant, which can be expressed as w(t):

[0081]

[0082] This gradient shares the same dimension as z, which means it can update z in pixels.

[0083] However, this will update both the foreground and background simultaneously, resulting in an inconsistent background.

[0084] To solve this problem, apply bbox B to the above formula. s To mask background-dependent gradients before backpropagation and obtain the background music of this invention:

[0085]

[0086] This simple masking strategy prevents the background from being updated, thus ensuring background consistency.

[0087] 3. Semantic Enhancement Concept Representation

[0088] The BGM module maintains background consistency during the swapping process. However, it cannot guarantee that the source concept can be replaced with the target concept. This limitation arises because BGM optimization still operates within the source latent z-value. t and target potential z t This is performed at the entire feature map level, without distinguishing between foreground and background. To address this issue, it is proposed to obtain semantically enhanced concept representations (SECRs) for both source and target concepts during concept exchange, emphasizing their positions within the foreground region.

[0089] Let F s p represents the source image features. s Hints representing source concepts, semantic embeddings s It can be accessed via c s =τ(p s )get.

[0090] First, adjust the size of the previously obtained object's bounding box to fit the source image feature F. s The size is used to obtain the feature bbo. x B f Then use B f Cut F s To obtain region image features f s Using f s via Q s =Wq·f s Calculate the query vector. Then, the key and value vectors can be obtained as follows:

[0091] K s =W k ·c s V s =W v ·c s · (6)

[0092] The final partial attention output is then calculated as follows:

[0093]

[0094] Here, d′ represents the output dimension of the key and value features. In this way, the semantic information of the source concept is injected into the cross-attention graph to obtain the region concept representation. Then Mapping back to the original feature map F s This yields the semantically enhanced representation of the entire source image.

[0095] By proactively injecting semantic guidance, SECR provides semantically enhanced conceptual representations for both source and target branches within the foreground region. Therefore, SECR transforms the target branch into a target concept adder and the source branch into a source concept deleter. Their collaboration enables precise and seamless concept exchange, thereby enhancing foreground consistency.

[0096] 4. Skip-step gradient update

[0097] Having resolved the inconsistency issue, this invention turns its attention to the challenge of inefficiency. For example... Figure 3 As shown, previous methods computed gradients at each time step.

[0098] During concept exchange, it was observed that the impact of gradient updates on the target image is similar across adjacent time steps. Based on this observation, a skip-step gradient update (SSGU) strategy is proposed. The key idea of ​​SSGU is that skipping some gradient calculations does not significantly sacrifice exchange consistency while significantly improving efficiency. Therefore, the SSGU of this invention calculates gradients at interval time steps and reuses previously calculated gradients during intermediate time steps.

[0099] Given source image X s Time step t and intermediate target image at time step t Define the entire pipeline as The gradient g at time step t can be obtained. t and output intermediate target image As shown below:

[0100]

[0101] Where η is the learning rate. The SSGU of this invention periodically retains some anchor gradients and skips the forward pass between two anchor gradients. The skip period is controlled by the SSGU factor λ. The anchor gradient set can be defined as:

[0102]

[0103] Where T is the final time step. For any intermediate time step t, the intermediate target image is updated using the gradient of its nearest previous anchor point. Taking λ = 2 as an example, assume t is even and g t-2 , SSGU uses anchor gradient g t-2 renew and

[0104]

[0105]

[0106]

[0107] For the next time step t, the gradient at another anchor point g t Used to update x t .

[0108] Therefore, the SSGU of the present invention reduces the number of forward passes in the entire concept exchange process to 1 / λ of the original count.

[0109] Since forward propagation accounts for approximately 95% of the total inference time, the SSGU of this invention can improve the overall inference speed of the method by approximately λ times with minimal impact on exchange consistency. Furthermore, the SSGU of this invention can be transferred to other fractional distillation-based methods to improve their efficiency in the same way.

[0110] experiment

[0111] Experiments were conducted on a single RTX 3090 using Stable Diffusion v2.1-base. A custom checkpointing mechanism from DreamBooth was used to introduce the concept.

[0112] SSG U factor λ The learning rate is set to 5, α to 2, β to 0.5, and the guidance scale to 7.5. The bounding box is obtained through the first three steps. Subsequently, SGD with a learning rate of 0.1 is used for optimization, iterating for 550 steps.

[0113] Qualitative comparison

[0114] Since custom concept exchange is a relatively new task, there are limited methods available for direct comparison. Therefore, state-of-the-art image editing methods are included and adapted for custom concept exchange. These methods include: (1) fractional distillation-based methods: SDS, DDS, and CDS; and (2) attention-based methods: PhotoSwap, PnPInv, and P2P. SwapAnything is excluded because it is not publicly available.

[0115] Qualitative comparison results in Figure 4 It is displayed in the middle.

[0116] Fractional distillation-based methods were found to be adaptable to shape changes during concept exchange. However, they exhibit poor foreground fidelity (lines 3 and 6) and lead to unnecessary modifications to the background (lines 1 and 2). Attention-based methods fail to manage shape changes (line 4) and struggle to maintain background consistency (line 5). In contrast, the method of this invention demonstrates superior performance in addressing shape changes and maintaining exchange consistency.

[0117] Quantitative comparison

[0118] A thorough quantitative comparison was also conducted. These methods were evaluated in three aspects using seven different metrics: (1) Foreground consistency: CLIP image scores were calculated between the generated image foreground and the custom concept image. (2) Background consistency: Background consistency was evaluated using four metrics: PSNR, LPIPS, MSE, and SSIM. (3) Overall consistency and efficiency: Overall cue consistency was evaluated by calculating CLIP text scores between the generated image and the target cue. The inference time for each method was also reported to assess its efficiency.

[0119] Table 1

[0120]

[0121] Table 1 shows that the method of the present invention outperforms all compared methods across all selected metrics. Red represents the best result, and blue represents the second best result.

[0122] Transitional Scheme Analysis

[0123] To verify the effectiveness of background music in preserving the background, an ablation study was conducted by deleting the background music.

[0124] Figure 5 The left side shows the qualitative results of the ablation study in this invention embodiment, while the right side shows the effects of applying BGM and SECR, and the right side shows the effects of using different bounding box generation methods. For example... Figure 5 As shown in the second column, while the method of this invention can still achieve concept exchange without background music, it severely modifies the background. In contrast, the complete method of this invention not only maintains high foreground fidelity but also effectively preserves background consistency.

[0125] To verify the effectiveness of SECR, ablation studies were conducted, including removing SECR from (1) the source branch (no source), (2) the target branch (no target), and (3) both (no source & target). Visualization results are shown in... Figure 5 In columns 3 through 5. While all methods preserve the background well, their foreground fidelity is reduced.

[0126] To verify the effectiveness of the proposed SSGU, images generated under different SSGU factors were first visualized. Figure 6 Showing different SSGU factors λ The qualitative result. Too high. λ This can lead to a decrease in the consistency of the prospect. For example... Figure 6 As shown, λ This indicates that SSGU is not being used. λ≤9In this case, SSGU can maintain the consistency between the foreground and background well, while improving the efficiency of the method.

[0127] In summary, this invention proposes an innovative customized concept replacement method, InstantSwap, which integrates the techniques of stable diffusion and fractional distillation to achieve efficient concept replacement without training and can effectively replace concepts across significant shape differences. The method first uses an automatic bounding box generation step to accurately locate the source concept in the image and generate the corresponding bounding box. Next, a background gradient masking (BGM) strategy is used to maintain background consistency while preventing unnecessary modifications to the background. Furthermore, semantically enhanced concept representation (SECR) injects semantic information into the foreground region defined by the bounding box, enhancing the representation of both the source and target concepts and improving the consistency of the foreground region in concept replacement. To further improve efficiency, this invention introduces a skip gradient update (SSGU) strategy. This strategy selectively performs forward pass gradient calculations during concept replacement and reuses previously calculated gradients in time steps where direct gradient calculations are not performed, thereby reducing the total number of forward passes and significantly improving the efficiency of the replacement process. This strategy not only reduces redundant computation but also plays a crucial role in maintaining the visual coherence and semantic consistency of the image.

[0128] Compared to existing technologies, this invention effectively overcomes a series of pain points in traditional concept replacement methods through its innovative InstantSwap framework. Addressing the inconsistency problem, this invention uses bounding boxes instead of traditional masking techniques, providing greater flexibility for shape changes during the concept replacement process. Simultaneously, it utilizes bounding boxes and gradient masks to prevent background changes, ensuring background consistency. Furthermore, by injecting semantic information to enhance concept representation, this invention improves foreground consistency, resolving the inconsistency between foreground and background in previous methods. To address inefficiency, this invention introduces a Skip Gradient Update (SSGU) strategy, performing forward gradient calculations only at specific time steps and reusing previously calculated gradients at other time steps, significantly reducing computational load and improving inference efficiency.

[0129] These innovative improvements of the present invention not only enhance the accuracy and efficiency of concept substitution but also expand its application value in multiple fields. In the field of selfie enhancement, users can replace specific concepts in photos according to their preferences to achieve personalized enhancement effects. Photo bloggers can use this technology to transform ordinary photos into images with specific themes or styles, adding creativity to their blog content. Comic artists can use CCS technology to quickly convert sketches or simple images into detailed and stylistically consistent comic art, improving creative efficiency. In advertising production, advertising companies can customize advertising visuals for brands, flexibly applying brand elements to different scenes and backgrounds. In film and animation production, CCS technology can be used to quickly replace and adjust scenes and characters, accelerating the production process. Online retailers can replace the backgrounds in product display images to create diverse visual effects and enhance the consumer shopping experience. In the field of education, by replacing the background environment of teaching materials and aids, it is possible to adapt to different cultural or curriculum needs, enhancing learning participation and interactivity. These application scenarios demonstrate the significant value of the present invention in improving the practicality and flexibility of concept substitution technology.

[0130] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0131] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0132] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0133] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0134] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0135] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0136] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0137] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0139] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0140] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0141] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0142] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A customized concept replacement method based on steady diffusion and fractional distillation, characterized in that, The method comprises the following steps: S1. Automatic bounding box generation: using a U-Net network to perform feature extraction on the source image to generate an image feature map; for the source image and the target image, generating noise prediction according to the respective text prompts; Using a cross-attention mechanism, combining the source image features and the text prompts of the source concept, to determine the location of the source concept in the image; according to the determined location, automatically generating a bounding box bbox indicating the source concept; S2. Background gradient mask BGM: using the generated bounding box bbox to define the foreground and background regions; using the latent features of the source image and the target image, calculating the gradient to reflect the difference between the latent features; Applying a gradient mask to the background region outside the foreground region to prevent modifying the background region during the concept replacement process to achieve background consistency information enhancement; S3. Semantically enhanced concept representation: within the foreground region defined by the bounding box bbox, injecting the semantic embedding obtained by combining the foreground region features and the text prompts into the cross-attention map of the source concept and the target concept, using semantic information to enhance the representation of the source concept and the target concept to improve the consistency of the foreground region in the concept replacement; S4. Concept replacement execution: using the source concept location determined by the bounding box bbox generated in step S1, the enhanced background consistency information in step S2, and the enhanced foreground consistency information in step S3, to perform the concept replacement process.

2. The customized concept replacement method based on steady diffusion and fractional distillation of claim 1, wherein, Step S1 specifically includes: Using a U-Net network to perform forward propagation on the source image to obtain image features; Combining the image features and the text prompts of the source concept to calculate a cross-attention map and a self-attention map; Modifying the cross-attention map by element power multiplication to strengthen the activation of the target region; Applying a threshold operation to obtain a coarse-grained mask; Converting the coarse-grained mask to an accurate bounding box to indicate the location of the source concept in the source image.

3. Customized conceptual replacement method based on stable diffusion and fractional distillation according to any of claims 1 to 2, characterized in that, In step S2, using the generated noise prediction and the automatically obtained bounding box bbox of the source concept, perform background gradient mask BGM to maintain background consistency.

4. The customized conceptual replacement method based on steady diffusion and fractional distillation according to any one of claims 1 to 2, characterized in that, Step S2 specifically includes: Combining the calculated gradient with the bounding box bbox of the source image to generate a background gradient mask BGM through element multiplication; Applying BGM to mask the background-related gradient to ensure that gradient updates only occur within the foreground region.

5. The customized concept replacement method based on steady diffusion and fractional distillation of claim 4, wherein, When calculating the gradient, a simplified model is applied to ignore the Jacobian of the U-Net, simplifying the gradient calculation process to a constant multiplied by the difference in latent features.

6. A customized conceptual replacement method based on steady diffusion and fractional distillation according to any one of claims 1 to 2, characterized in that, Step S3 specifically includes: Adjusting the size of the previously obtained bounding box bbox to adapt to the size of the source image features to form a feature bounding box feature bbox; Using the feature bounding box to crop the source image features to extract the region image features related to the source concept; Using the region image features and the text prompts of the source concept to calculate a query vector; According to the query vector and the semantic embedding of the source concept, calculating a key vector and a value vector; Through the cross-attention mechanism, injecting the semantic information of the source concept into the region image features to obtain a region concept representation; The region concept representation is mapped back to the original feature map to obtain a semantic enhanced representation of the entire source image. The semantic enhanced concept representation is used to achieve precise and seamless concept replacement in the foreground region to enhance the consistency of the foreground.

7. A customized conceptual replacement method based on steady diffusion and fractional distillation according to any one of claims 1 to 2, characterized in that, In step S4, the skip-grad step update is performed: in the process of concept replacement, the gradient is calculated by forward propagation at a given time step; at the time step where direct gradient calculation is not performed, the previously calculated gradient is reused for parameter update; by skipping the gradient calculation of non-critical time steps, the total number of forward propagation is reduced, and the efficiency of the replacement process is improved.

8. The customized concept replacement method based on steady diffusion and fractional distillation of claim 7, wherein, Step S4 specifically includes: Defining a calculation model for the entire concept replacement process, taking the source image, the intermediate target image and the time step as input; Performing forward propagation at a given time step to calculate the gradient update; Determine the skip period for skipping gradient calculation at adjacent time steps, which is controlled by a preset skip gradient update SSGU factor; At the time step where direct gradient calculation is not performed, the anchor gradient calculated at the nearest time step is reused for parameter update.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, implements the customized concept replacement method based on stable diffusion and fractional distillation as claimed in any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the customized concept replacement method based on stable diffusion and fractional distillation as claimed in any one of claims 1 to 8.

Images