Customized concept replacement method based on stable diffusion and fractional distillation

Abstract

The invention provides a high-efficiency customized concept replacement method, and the method achieves the immediate exchange, and can process the shape difference between a source concept and a target concept while maintaining the consistency of a foreground and a background without training. The method includes automatically generating bounding boxes to locate source concepts, applying a background gradient mask to protect a background region, enhancing foreground consistency by semantically enhancing concept characterization, and performing concept permutation. Furthermore, a leapfrogging gradient updating SSGU strategy is introduced, the gradient is calculated only in a key time step, the number of times of forward transmission is reduced, and the efficiency is improved. Through wide qualitative and quantitative evaluation and comprehensive ablation research, the effectiveness and superiority of the method are proved, and the multifunctionality and wide applicability of the method in related tasks are shown. The method is particularly suitable for application scenes needing accurate concept replacement, such as selfie beautification, photo blog and cartoon creation and the like.

Description

Technical Field

[0001] The present invention relates to image editing technology, and in particular to a customized concept replacement method based on stable diffusion and fractional distillation. Background Art

[0002] Image editing is a fundamental and popular topic in computer vision. Previous studies based on generative adversarial networks (GANs) have focused only on specific object domains, which has limited their applications. With the advent of diffusion models, image editing is now able to modify a variety of objects via cues. These methods can be mainly divided into five categories: instruction-based methods, hybrid-based methods, attention-based methods, inversion-based methods, and fractional distillation-based methods. Instruction-based methods (instructpix2pix, instructdiffusion, smart edit, magic brush) usually require instruction editing datasets to train diffusion models. Hybrid-based methods merge source and target cues to guide the editing process, while attention-based methods inject attention features of the source image. Both methods have low editing costs but poor background preservation and cue alignment. Inversion-based methods aim to invert the fixed trajectories generated by the forward pass to reproduce the source image. These methods can be used as an additional training stage to enhance the background consistency of attention-based methods. Finally, fractional distillation-based methods borrow the optimization process of and use a fractional distillation-based loss to optimize the source image for editing. These methods are more flexible than previous methods, but still face challenges in background preservation.

[0003] Customized Concept Substitution (CCS), a subtask of text-to-image (T2I) generation, aims to replace a concept in a source image with a highly customized new concept. Combined with diffusion models, recent customized concept substitution methods have shown wide applicability in areas such as selfie beautification, photo blog creation, and comics creation.

[0004] Early works on customized concept replacement mainly relied on copy-paste techniques, which are crude and unreliable. A series of research results have been proposed by combining powerful customization techniques with image editing methods. Despite their remarkable success, these methods still face the problems of inconsistency and inefficiency. Figure 1As shown: (1) Inconsistency: Attention-based methods such as PhotoSwap and P2P, although they can maintain background consistency well, perform poorly when dealing with shape differences between source and target concepts, resulting in foreground inconsistency. Score distillation-based methods, such as SDS, DDS, and CDS, fail to accurately generate foreground concepts and significantly change the background, resulting in double inconsistency between foreground and background. (2) Inefficiency: Attention-based methods require an inefficient training phase on the source image to maintain background consistency. Although score distillation-based methods do not require training, they still require redundant forward pass calculations at each time step, resulting in inefficient inference.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide a customized concept replacement method based on stable diffusion and fractional distillation.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A customized concept replacement method based on stable diffusion and fractional distillation, comprising the following steps:

[0009] S1. Automatic bounding box generation: Extract features from the source image and generate an image feature map; use the cross-attention mechanism to combine the source image features and the textual hints of the source concept to determine the location of the source concept in the image; based on the determined location, automatically generate a bounding box bbox indicating the source concept;

[0010] S2. Background gradient mask BGM: Use the generated bounding box bbox to define the foreground and background areas; apply a gradient mask to the background area outside the foreground area to prevent the background area from being modified during the concept replacement process;

[0011] S3. Semantically enhanced concept representation: In the foreground region defined by the bounding box bbox, the semantic embedding obtained by combining the foreground region features and textual cues is injected into the cross-attention map of the source concept and the target concept, and the representation of the source concept and the target concept is enhanced using semantic information to improve the consistency of the foreground region in concept replacement;

[0012] S4. Concept replacement execution: The concept replacement process is performed using the source concept location determined by the bounding box bbox generated in step S1 and the background and foreground consistency information enhanced in steps S2 and S3.

[0013] Furthermore, in step S1, a U-Net network is used to extract features from the source image.

[0014] Furthermore, step S1 specifically includes:

[0015] Use the U-Net network to perform forward pass on the source image to obtain image features;

[0016] Combine image features and textual hints of source concepts to compute cross-attention maps and self-attention maps;

[0017] Modify the cross-attention map to strengthen the activation of the target area through element-wise power multiplication;

[0018] Apply a threshold operation to obtain a coarse-grained mask;

[0019] The coarse-grained mask is converted into a precise bounding box to indicate the location of the source concept in the source image.

[0020] Furthermore, in step S1, for the source image and the target image, noise predictions are generated according to their respective text prompts; in step S2, background gradient masking (BGM) is performed using the generated noise predictions and the automatically acquired bounding box bbox of the source concept to maintain background consistency.

[0021] Furthermore, step S2 specifically includes:

[0022] Using the potential features of the source image and the target image, the gradient is calculated to reflect the difference between the potential features; preferably, when calculating the gradient, a simplified model is applied to ignore the Jacobian term of the U-Net, and the gradient calculation process is simplified to a form of a constant multiplied by the potential feature difference;

[0023] Combine the calculated gradient with the bounding box bbox of the source image and generate the background gradient mask BGM through element-wise multiplication operation;

[0024] BGM is applied to mask the gradients related to the background, ensuring that the gradient update only occurs in the foreground area.

[0025] Furthermore, step S3 specifically includes:

[0026] Adjust the size of the previously obtained bounding box bbox to adapt to the size of the source image feature to form the feature bounding box feature bbox;

[0027] Using feature bounding boxes to crop source image features to extract regional image features related to source concepts;

[0028] Compute the query vector using regional image features and textual hints of source concepts;

[0029] Calculate the key vector and value vector based on the query vector and the semantic embedding of the source concept;

[0030] Through the cross-attention mechanism, the semantic information of the source concept is injected into the regional image features to obtain the regional concept representation;

[0031] The regional concept representation is mapped back to the original feature map to obtain a semantically enhanced representation of the entire source image. The semantically enhanced concept representation is used to achieve precise and seamless concept replacement in the foreground region to enhance the consistency of the foreground.

[0032] Furthermore, in step S4, a skip-step gradient update is performed: in the process of concept replacement, a given time step is selected to perform a forward pass to calculate the gradient; at the time step where direct gradient calculation is not performed, the previously calculated gradient is reused for parameter update; by skipping the gradient calculation of non-critical time steps, the total number of forward passes is reduced, thereby improving the efficiency of the replacement process.

[0033] Furthermore, step S4 specifically includes:

[0034] Define a computational model for the entire concept permutation process, taking as input the source image, the intermediate target image, and the time step;

[0035] Perform a forward pass at a given time step, computing the gradient update;

[0036] Determine the skipping period, which is used to skip the gradient calculation on adjacent time steps. The skipping period is controlled by the preset SSGU factor;

[0037] At time steps where direct gradient computation is not performed, the anchor gradients computed at the most recent time step are reused for parameter updates.

[0038] A computer-readable storage medium stores a computer program, characterized in that when the computer program is executed by a processor, it implements the customized concept replacement method based on stable diffusion and fractional distillation.

[0039] A computer program product includes a computer program, characterized in that when the computer program is executed by a processor, it implements the customized concept replacement method based on stable diffusion and fractional distillation.

[0040] The present invention has the following beneficial effects:

[0041] The present invention provides a customized concept replacement method based on stable diffusion and fractional distillation, and proposes an innovative customized concept replacement framework without training, called InstantSwap, which can effectively perform efficient concept replacement while maintaining the consistency of foreground and background. The framework accurately locates the source concept through automatic bounding box generation, and uses the background gradient mask (BGM) strategy to prevent unnecessary modification of the background and ensure the stability of the background. At the same time, the semantic information of the foreground area is enhanced by semantic enhanced concept representation (SECR), which improves the consistency and accuracy of foreground replacement. Furthermore, the present invention also introduces a step jump gradient update (SSGU) strategy, which significantly reduces the total number of forward passes by performing forward pass calculations at specific time steps and reusing previously calculated gradients at other time steps, thereby improving the overall reasoning efficiency. The method of the present invention not only reduces redundant calculations, but also maintains the visual coherence and semantic consistency of the image during the concept replacement process, so that customized concept replacement has a wide range of applicability in various application scenarios such as selfie beautification, photo blog creation and comic creation. Through extensive qualitative and quantitative evaluations, as well as comprehensive ablation studies, we demonstrate the effectiveness and superiority of our approach, showcasing its versatility and wide applicability in relevant tasks.

[0042] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A comparison diagram showing that the method according to the embodiment of the present invention achieves better exchange consistency than the existing method.

[0044] Figure 2 The figure is an overall flow chart of a customized concept replacement method based on stable diffusion and fractional distillation according to an embodiment of the present invention.

[0045] Figure 3 4 is a comparison diagram between the SSGU of the embodiment of the present invention and the previous method.

[0046] Figure 4 4 is a qualitative comparison result diagram of the embodiments of the present invention.

[0047] Figure 5 This is a qualitative result of the ablation study of an embodiment of the present invention. The left side shows the effect of applying BGM and SECR, and the right side shows the effect of using different bbox generation methods.

[0048] Figure 6 Different SSGU factors of the embodiments of the present invention λ The qualitative results are too high. λ This will lead to a decrease in outlook consistency. DETAILED DESCRIPTION

[0049] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0050] Concept swapping is a subtask of general image editing, which mainly replaces the source concept in the image with the target concept specified by the user. The task was first proposed by PbE, which used the CLIP encoder to extract the features of the target concept and injected it into the UNet through the cross-attention layer. Since then, parallel work has extended concept swapping to the customization domain. They combine the attention-based editing method with the adjustment-based customization method to achieve customized concept swapping. Based on Photoswap, SwapAnything further uses an external module to obtain a mask to specify the location of the object in the source image. The embodiments of the present invention improve the existing methods in several important aspects. First, a bounding box is used instead of a mask as a spatial indicator of the source concept, thereby providing greater flexibility for shape changes during concept swapping. Second, a bounding box is used to prevent background changes through a gradient mask, thereby ensuring background consistency. Third, the concept representation is enhanced with semantic information to maintain foreground consistency. Finally, instead of performing forward passes at every time step, they are only performed at specific intervals to improve efficiency.

[0051] The embodiment of the present invention provides a customized concept replacement method based on stable diffusion and fractional distillation, comprising the following steps:

[0052] Step S1. Automatic bounding box generation: extract features from the source image and generate an image feature map; use the cross-attention mechanism to combine the source image features and the textual prompts of the source concept to determine the location of the source concept in the image; based on the determined location, automatically generate a bounding box bbox indicating the source concept.

[0053] In a preferred embodiment, a U-Net network is used in step S1 to extract features from the source image. Further, step S1 specifically includes: performing a forward pass on the source image using the U-Net network to obtain image features; combining image features and textual hints of source concepts to calculate a cross-attention map and a self-attention map; modifying the cross-attention map to strengthen the activation of the target area through element-wise power multiplication operations; applying a threshold operation to obtain a coarse-grained mask; and converting the coarse-grained mask into a precise bounding box to indicate the location of the source concept in the source image.

[0054] Step S2. Background Gradient Masking (BGM): Use the generated bounding box bbox to define the foreground and background areas; apply a gradient mask to the background area outside the foreground area to prevent the background area from being modified during the concept replacement process.

[0055] In a preferred embodiment, step S2 specifically includes: using the potential features of the source image and the target image to calculate the gradient to reflect the difference between the potential features; preferably, when calculating the gradient, a simplified model is applied to ignore the Jacobian term of U-Net, and the gradient calculation process is simplified to the form of a constant multiplied by the potential feature difference; the calculated gradient is combined with the bounding box bbox of the source image, and a background gradient mask BGM is generated through an element-wise multiplication operation; BGM is applied to shield the gradient related to the background to ensure that the gradient update occurs only in the foreground area.

[0056] In a preferred embodiment, in step S1, for the source image and the target image, noise predictions are generated according to their respective text prompts; in step S2, background gradient masking (BGM) is performed using the generated noise predictions and the automatically acquired bounding box bbox of the source concept to maintain background consistency.

[0057] Step S3. Semantic Enhanced Concept Representation (SECR): In the foreground region defined by the bounding box bbox, the semantic embedding obtained by combining the foreground region features and textual cues is injected into the cross-attention map of the source concept and the target concept, and the representation of the source concept and the target concept is enhanced using semantic information to improve the consistency of the foreground region in concept permutation.

[0058] In a preferred embodiment, step S3 specifically includes: adjusting the size of the previously obtained bounding box bbox to adapt to the size of the source image feature to form a feature bounding box feature bbox; using the feature bounding box to crop the source image feature to extract the regional image feature related to the source concept; using the regional image feature and the text prompt of the source concept to calculate the query vector; calculating the key vector and the value vector according to the semantic embedding of the query vector and the source concept; injecting the semantic information of the source concept into the regional image feature through the cross-attention mechanism to obtain the regional concept representation; mapping the regional concept representation back to the original feature map to obtain the semantically enhanced representation of the entire source image, and using the semantically enhanced concept representation to achieve accurate and seamless concept replacement in the foreground area to enhance the consistency of the foreground.

[0059] Step S4. Concept replacement execution: The concept replacement process is performed using the source concept location determined by the bounding box (bbox) generated in step S1, and the background and foreground consistency information enhanced in steps S2 and S3.

[0060] In a preferred embodiment, in step S4, a step-skipping gradient update (SSGU) is performed: in the process of concept replacement, a given time step is selected to perform a forward pass to calculate the gradient; at the time step where direct gradient calculation is not performed, the previously calculated gradient is reused for parameter update; by skipping the gradient calculation of non-critical time steps, the total number of forward passes is reduced, thereby improving the efficiency of the replacement process.

[0061] Furthermore, the skipping gradient update method of step S4 specifically includes: defining a computational model of the entire concept replacement process, taking the source image, the intermediate target image and the time step as input; performing a forward pass at a given time step to calculate the gradient update; determining a skipping period for skipping gradient calculation at adjacent time steps, wherein the skipping period is controlled by a preset SSGU factor; at the time step where direct gradient calculation is not performed, reusing the anchor gradient calculated at the most recent time step for parameter update.

[0062] The embodiment of the present invention provides a customized concept replacement method based on stable diffusion and fractional distillation, and proposes InstantSwap, which is a training-free framework that can efficiently perform customized concept replacement in shape differences while maintaining the consistency of foreground and background. Specifically, the present invention extracts a bounding box (bbox) indicating the location of the source concept from the enhanced cross-attention map of the source image. Using this bbox, a background gradient mask (BGM) strategy is implemented to prevent modifications to the parts outside the bbox, thereby ensuring background consistency. In addition, in order to improve foreground consistency, semantic information is used to highlight the cross-attention maps of the source concept and the target concept in the bbox respectively. This strategy leads to semantically enhanced concept representation (SECR), which contributes to accurate foreground replacement. Further, a step skipping gradient update (SSGU) strategy is introduced to perform a forward pass only at a specific time step to calculate the gradient. For time steps without direct gradient calculation, the previously obtained gradient is reused for updating. Through this strategy, the total number of forward passes is reduced, and the efficiency of the method of the present invention is improved.

[0063] Through extensive qualitative and quantitative comparisons, the effectiveness and superiority of our approach are demonstrated. Comprehensive ablation studies validate the effectiveness of each component in our approach. In addition, we further extend InstantSwap to related tasks, demonstrating its efficacy and versatility.

[0064] The following further describes specific embodiments of the present invention, algorithm examples and experimental verification.

[0065] Figure 2The overall process of the customized concept replacement method based on stable diffusion and fractional distillation is shown.

[0066] The embodiment of the present invention proposes a novel training-free customized concept exchange (CCS) framework, which can effectively exchange concepts across obvious shape differences. A background gradient masking (BGM) strategy and a semantic enhanced concept representation (SECR) are designed to improve the consistency of background and foreground respectively. Furthermore, a step skip gradient update (SSGU) strategy is used to reduce redundant calculations and improve efficiency.

[0067] In some embodiments, the bbox of the source concept is first automatically obtained. The obtained bbox is input into SECR in the source and target branches to enhance the foreground swap consistency. In some embodiments, the source and target branches generate noise predictions for the source and target images according to their respective cues. The predicted noise is used together with the bbox for BGM to maintain background consistency.

[0068] 1. Automatic bounding box generation

[0069] First, automatically obtain the bbox to indicate the concept O s Position in the source image.

[0070] Given a source image x s and source prompt P s , using U-Net∈ φ (·) Perform a forward pass and obtain the cross-attention map A by c and self-attention map A s :

[0071]

[0072] Where Q is the query vector projected from the image features and d represents the output dimension of the key and query features.

[0073] K is the key vector and V is the value vector. For the cross attention graph A c , K and V are projected from the text embedding τ(P). For the self-attention map A s , V and V are projected from image features. Applying threshold directly on SA^c$ will produce a coarse-grained mask that cannot accurately reflect O s Position. c The following modifications were made:

[0074]

[0075] According to formula 1, A c All values ​​in are between 0 and 1. Therefore, for A c Element-wise multiplication by α can weaken the activation of non-target areas. In addition, As Contains rich structural information. This information can effectively help Finally, the threshold β is applied to To obtain the mask. Then, the mask is converted to bbox B according to the minimum and maximum coordinates of all foreground points within the mask. s This strategy enables to obtain bbo without adding any modules x B s A relatively loose constraint is intentionally set on the mask to obtain a bbox that fully covers the source concept.

[0076] 2. Background Gradient Mask

[0077] Using the object bbox, a background gradient masking (BGM) method is proposed to ensure that the concept exchange is confined to the foreground area. Given the latent features of the source image and the latent feature z of the target image, where z is initialized as And continuously optimize to obtain the final target image x t .

[0078] Based on DDS, first obtain the gradient of z:

[0079]

[0080] As described in SDS, the intermediate term is the U-Net Jacobian term and can be omitted, and is a constant and can be expressed as w(t):

[0081]

[0082] This gradient shares the same dimensions as z, which means it can update z pixel-wise.

[0083] However, this will update both the foreground and the background simultaneously, resulting in an inconsistent background.

[0084] To solve this problem, apply bbox B to the above formula s To shield the gradient related to the background before back-propagation and obtain the BGM of the present invention:

[0085]

[0086] This simple masking strategy prevents the background from being updated, thus ensuring background consistency.

[0087] 3. Semantic Enhancement of Concept Representation

[0088] The BGM module maintains the consistency of the background during the exchange process. However, there is no guarantee whether the source concept can be replaced by the target concept. This limitation occurs because the optimization of the BGM is still based on the source potential z t and target potential z t The whole feature map level is performed without distinguishing the foreground and background. To address this issue, we propose to obtain semantically enhanced concept representations (SECR) for both source and target concepts during concept exchange and emphasize their locations within the foreground region.

[0089] Let F s is the source image feature, p s Hint representing the source concept, semantic embedding c s Available through c s =τ(p s )get.

[0090] First, resize the previously obtained object bbox to fit the source image feature F s The size of the feature bbo is obtained x B f Then use B f Crop F s To obtain the regional image feature f s . Using f s , via Q s =Wq·f s Calculate the query vector. Afterwards, you can get the key and value vectors as follows:

[0091] K s =W k ·c s , V s =W v ·c s · (6)

[0092] The final partial attention output is then calculated as follows:

[0093]

[0094] Where d′ represents the output dimension of key and value features. In this way, the semantic information of the source concept is injected into the cross attention map to obtain the regional concept representation Then Mapping back to the original feature map F s , get the semantically enhanced representation of the entire source image

[0095] By actively injecting semantic guidance, we provide semantically enhanced concept representations for both source and target branches within the foreground region. Therefore, SECR transforms the target branch into a target concept adder and the source branch into a source concept remover. Their collaboration enables precise and seamless concept exchange, thus enhancing foreground consistency.

[0096] 4. Skip-step gradient update

[0097] After solving the inconsistency problem, the present invention turns its attention to the challenge of inefficiency. Figure 3 As shown, previous methods compute the gradient at each time step.

[0098] During concept exchange, it is observed that the impact of gradient updates on the target image is similar at adjacent time steps. Based on this observation, a stepped skip gradient update (SSGU) strategy is proposed. The key idea of ​​SSGU is that skipping some gradient calculations does not significantly sacrifice exchange consistency while significantly improving efficiency. Therefore, the SSGU of the present invention calculates gradients at interval time steps and reuses previously calculated gradients during intermediate time steps.

[0099] Given a source image X s , time step t and the intermediate target image at time step t The entire pipeline is defined as The gradient g at time step t can be obtained t And output intermediate target image As shown below:

[0100]

[0101] Where η is the learning rate. The SSGU of the present invention periodically retains some anchor gradients and skips the forward pass between two anchor gradients. The skipping period is controlled by the SSGU factor λ. The anchor gradient set can be defined as:

[0102]

[0103] where T is the ending time step. For any intermediate time step t, the intermediate target image is updated using its nearest previous anchor gradient Taking λ=2 as an example, assuming that t is an even number and g t-2 , SSGU uses anchor gradient g t-2 renew and

[0104]

[0105]

[0106]

[0107] For the next time step t, another anchor gradient g t To update x t .

[0108] Therefore, the SSGU of the present invention reduces the number of forward passes in the entire concept exchange process to 1 / λ of the original count.

[0109] Since the forward pass accounts for about 95% of the total inference time, the SSGU of the present invention can improve the overall inference speed of the method by about λ times with minimal impact on exchange consistency. In addition, the SSGU of the present invention can be transferred to other fractional distillation-based methods to improve their efficiency in the same way.

[0110] experiment

[0111] Experiments using Stable Diffusion v2.1-base on a single RTX3090. Used DreamBooth's custom checkpoints to introduce concepts.

[0112] SSG U factor λ is set to 5, α is set to 2, β is set to 0.5, and the guidance scale is set to 7.5. The bbox is obtained through the first three steps. Subsequently, SGD with a learning rate of 0.1 is used for optimization and iterates for 550 steps.

[0113] Qualitative comparison

[0114] Since custom concept swapping is a relatively new task, there are limited methods available for direct comparison. Therefore, we include SOTA image editing methods and adapt them to custom concept swapping. The following methods are included: (1) fractional distillation based methods: SDS, DDS, and CDS; (2) attention based methods: PhotoSwap, PnPInv, and P2P. SwapAnything is excluded because it is not publicly available.

[0115] The qualitative comparison results are Figure 4 Displayed in.

[0116] It is found that fractional distillation based methods can adapt to shape changes during concept exchange. However, they have poor foreground fidelity (rows 3 and 6) and cause unnecessary modifications to the background (rows 1 and 2). Attention-based methods cannot manage shape changes (row 4) and have difficulty maintaining background consistency (row 5). In contrast, our method shows excellent performance in addressing shape changes and maintaining exchange consistency.

[0117] Quantitative comparison

[0118] A thorough quantitative comparison is also performed. These methods are evaluated from three aspects using seven different metrics: (1) Foreground consistency: CLIP image score between the generated image foreground and the customized concept image is calculated. (2) Background consistency: Four metrics PSNR, LPIPS, MSE, SSIM are used to evaluate background consistency. (3) Overall consistency and efficiency: CLIP text score between the generated image and the target cue is calculated to evaluate the overall cue consistency. The inference time of each method is also reported to evaluate its efficiency.

[0119] Table 1

[0120]

[0121] Table 1 shows that the method of the present invention outperforms all the compared methods in all selected indicators. Red represents the best result and blue represents the second best result.

[0122] Transition plan analysis

[0123] To verify the effectiveness of BGM in background preservation, an ablation study was performed by removing the BGM.

[0124] Figure 5 The following are qualitative results of the ablation study of the embodiment of the present invention. The left side shows the effect of applying BGM and SECR, and the right side shows the effect of using different bbox generation methods. Figure 5 As shown in the second column, although the proposed method can still achieve concept exchange without BGM, it will cause serious modifications to the background. In contrast, the proposed method not only maintains high foreground fidelity, but also effectively preserves background consistency.

[0125] To verify the effectiveness of SECR, an ablation study is performed, including removing SECR from (1) the source branch (no source), (2) the target branch (no target), and (3) both (no source & target). The visualization results are shown in Figure 5 3 to 5. Although all methods can preserve the background well, their foreground fidelity is degraded.

[0126] To verify the effectiveness of the proposed SSGU, we first visualize the images generated under different SSGU factors. Figure 6 Showing different SSGU factors λ The qualitative results are too high. λ This will lead to a decrease in foreground consistency. Figure 6 As shown, λ Indicates that SSGU is not used. λ≤9When , SSGU can well maintain the consistency of foreground and background while improving the efficiency of the method.

[0127] In summary, the present invention proposes an innovative customized concept replacement method, namely InstantSwap, which integrates the technical ideas of stable diffusion and fractional distillation, realizes efficient concept replacement without training, and can effectively replace concepts across obvious shape differences. The method first accurately locates the position of the source concept in the image through the automatic bounding box generation step and generates the corresponding bounding box. Then, the background gradient mask (BGM) strategy is used to prevent unnecessary modification of the background while maintaining the consistency of the background. In addition, semantically enhanced concept representation (SECR) is used to inject semantic information into the foreground area defined by the bounding box, which enhances the representation of the source concept and the target concept, thereby improving the consistency of the foreground area in concept replacement. In order to further improve the efficiency, the present invention introduces the skipped gradient update (SSGU) strategy, which selectively performs forward pass calculations in the concept replacement process and reuses the previously calculated gradients at the time step where direct gradient calculations are not performed, thereby reducing the total number of forward passes and significantly improving the efficiency of the replacement process. This strategy not only reduces redundant calculations, but also plays an important role in maintaining the visual coherence and semantic consistency of the image.

[0128] Compared with existing technologies, the present invention effectively overcomes a series of pain points in traditional concept replacement methods through an innovative technical framework, InstantSwap. To address the inconsistency problem, the present invention uses a bounding box instead of the traditional mask technology, which provides greater flexibility for shape changes during concept exchange. At the same time, the bounding box and gradient mask are used to prevent background changes and ensure the consistency of the background. In addition, by injecting semantic information to enhance the concept representation, the present invention improves the consistency of the foreground and solves the problem of inconsistency between foreground and background in previous methods. For the inefficiency problem, the present invention introduces a skipped gradient update (SSGU) strategy, which only performs a forward pass to calculate the gradient at a specific time step, and reuses the previously calculated gradient at other time steps, which significantly reduces the amount of calculation and improves the reasoning efficiency.

[0129] These innovative improvements of the present invention not only improve the accuracy and efficiency of concept replacement, but also expand its application value in multiple fields. In the field of selfie beautification, users can replace specific concepts in photos according to their preferences to achieve personalized beautification effects. Photo blog creators can use this technology to convert ordinary photos into images with specific themes or styles to add creativity to blog content. Comic artists can use CCS technology to quickly convert sketches or simple images into detailed and consistent comic art to improve creative efficiency. In advertising production, advertising companies can customize advertising images for brands and flexibly apply brand elements to different scenes and backgrounds. In the process of film and animation production, CCS technology can be used to quickly replace and adjust scenes and characters to speed up the production process. Online retailers can replace the background in product display pictures to create diverse visual effects and enhance consumers' shopping experience. In the field of education, by replacing the background environment of teaching materials and teaching aids, it can adapt to different cultural or course requirements and enhance the sense of participation and interactivity of learning. These application scenarios demonstrate the important value of the present invention in improving the practicality and flexibility of concept replacement technology.

[0130] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0131] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0132] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0133] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0134] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0135] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0137] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0138] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0139] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0140] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0141] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0142] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A customized concept replacement method based on stable diffusion and fractional distillation, characterized in that: The following steps are involved: S1. Automatic bounding box generation: Extract features from the source image and generate an image feature map; use the cross-attention mechanism to combine the source image features and the textual hints of the source concept to determine the location of the source concept in the image; based on the determined location, automatically generate a bounding box bbox indicating the source concept; S2. Background gradient mask BGM: Use the generated bounding box bbox to define the foreground and background areas; Applying a gradient mask to the background area outside the foreground area to prevent the background area from being modified during the concept replacement process; S3. Semantically enhanced concept representation: In the foreground region defined by the bounding box bbox, the semantic embedding obtained by combining the foreground region features and textual cues is injected into the cross-attention map of the source concept and the target concept, and the representation of the source concept and the target concept is enhanced using semantic information to improve the consistency of the foreground region in concept replacement; S4. Concept replacement execution: The concept replacement process is performed using the source concept location determined by the bounding box bbox generated in step S1 and the background and foreground consistency information enhanced in steps S2 and S3.

2. The customized concept replacement method based on stable diffusion and fractional distillation according to claim 1, characterized in that: In step S1, a U-Net network is used to extract features from the source image.

3. The customized concept replacement method based on stable diffusion and fractional distillation according to claim 2, characterized in that: Step S1 specifically includes: Use the U-Net network to perform forward pass on the source image to obtain image features; Combine image features and textual hints of source concepts to compute cross-attention maps and self-attention maps; Modify the cross-attention map to strengthen the activation of the target area through element-wise power multiplication; Apply a threshold operation to obtain a coarse-grained mask; The coarse-grained mask is converted into a precise bounding box to indicate the location of the source concept in the source image.

4. The customized concept replacement method based on stable diffusion and fractional distillation according to any one of claims 1 to 3, characterized in that: In step S1, for the source image and the target image, noise predictions are generated according to the respective textual cues; In step S2, a background gradient mask (BGM) is performed using the generated noise prediction and the automatically acquired bounding box (bbox) of the source concept to maintain background consistency.

5. The customized concept replacement method based on stable diffusion and fractional distillation according to any one of claims 1 to 4, characterized in that: Step S2 specifically includes: Using the potential features of the source image and the target image, the gradient is calculated to reflect the difference between the potential features; preferably, when calculating the gradient, a simplified model is applied to ignore the Jacobian term of the U-Net, and the gradient calculation process is simplified to a form of a constant multiplied by the potential feature difference; Combine the calculated gradient with the bounding box bbox of the source image and generate the background gradient mask BGM through element-wise multiplication operation; BGM is applied to mask the gradients related to the background, ensuring that the gradient update only occurs in the foreground area.

6. The customized concept replacement method based on stable diffusion and fractional distillation according to any one of claims 1 to 5, characterized in that: Step S3 specifically includes: Adjust the size of the previously obtained bounding box bbox to adapt to the size of the source image feature to form the feature bounding box feature bbox; Using feature bounding boxes to crop source image features to extract regional image features related to source concepts; Compute the query vector using regional image features and textual hints of source concepts; Calculate the key vector and value vector based on the query vector and the semantic embedding of the source concept; Through the cross-attention mechanism, the semantic information of the source concept is injected into the regional image features to obtain the regional concept representation; The regional concept representation is mapped back to the original feature map to obtain a semantically enhanced representation of the entire source image. The semantically enhanced concept representation is used to achieve precise and seamless concept replacement in the foreground region to enhance the consistency of the foreground.

7. The customized concept replacement method based on stable diffusion and fractional distillation according to any one of claims 1 to 6, characterized in that: In step S4, a skip-step gradient update is performed: in the process of concept replacement, a given time step is selected to perform a forward pass to calculate the gradient; at the time step where direct gradient calculation is not performed, the previously calculated gradient is reused for parameter update; by skipping the gradient calculation of non-critical time steps, the total number of forward passes is reduced, thereby improving the efficiency of the replacement process.

8. The customized concept replacement method based on stable diffusion and fractional distillation according to claim 7, characterized in that: Step S4 specifically includes: Define a computational model for the entire concept permutation process, taking as input the source image, the intermediate target image, and the time step; Perform a forward pass at a given time step, computing the gradient update; Determine the skipping period, which is used to skip the gradient calculation on adjacent time steps. The skipping period is controlled by the preset SSGU factor; At time steps where direct gradient computation is not performed, the anchor gradients computed at the most recent time step are reused for parameter updates.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the customized concept replacement method based on stable diffusion and fractional distillation as described in any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the customized concept replacement method based on stable diffusion and fractional distillation as described in any one of claims 1 to 8 is implemented.

Images