Style object insertion method based on feature injection

Abstract

The invention discloses a style object insertion method based on feature injection, belongs to the technical field of image editing, and solves the problem that an existing style object insertion method cannot seamlessly insert a specified object in a style image background. According to the method, an initial composite image is generated by using a local redrawing type diffusion model, and then stylized processing is carried out by combining noise inversion and feature injection technologies, so that a specified object is seamlessly inserted into a style background image; moreover, the method does not need to carry out model training for a specific style or object, remarkably improves the convenience and universality of operation, can effectively migrate the style features of the background image to the insertion object through mixing the content noise with the style noise, and injects the key value features of the style background in the denoising process, thereby improving the recognition efficiency of the background image. And the high consistency of the composite image in the visual style is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image editing, in particular to a style object insertion method based on feature injection. BACKGROUND

[0002] Image editing technology has wide application prospects in the fields of artistic creation, film special effects, virtual reality, etc. Style object insertion aims to solve a key problem in the field of computer vision and image generation: how to seamlessly integrate a specified object into a stylized image so that it is consistent with the original image in terms of visual style. Traditional image synthesis methods (such as cutout paste) are difficult to match the abstract texture of stylized images (such as oil painting, watercolor), and current style transfer methods are mostly aimed at global stylization and are difficult to control locally; while generation based on diffusion models can maintain style, but the position and shape accuracy of object insertion are insufficient. Style object insertion can stylize a specified object through an image generation model and achieve seamless insertion, significantly improving the naturalness of the synthesis effect. Through the research of style object insertion technology, the image editing experience of "what you think is what you get" can be realized, which has great research significance in the fields of artistic creation, virtual content production, advertising marketing, etc. Combined with style transfer, semantic segmentation, image inpainting, etc., it can promote the integration of multi-modal generation technology and solve the limitations of existing technologies.

[0003] Existing style object insertion methods, such as AnyDoor: Zero-shot Object-level Image Customization, use a generation model as the backbone, first extract multi-scale appearance and structural clues from the content image and style image, then redraw and stylize the target object in the latent space through attention or feature modulation, control the editing range with a local mask, and maintain the semantic shape and material consistency of the object during the gradual denoising process to achieve the goal of "preserving content semantics while injecting style features". However, this method mainly performs style injection within the object and lacks collaborative modeling of the background, which cannot effectively transfer the style elements of the background to the inserted object, resulting in style mismatch and poor fusion, and the inserted object cannot be well integrated with the background. SUMMARY

[0004] To solve the above problems in the prior art, the present application provides a style object insertion method based on feature injection, which solves the problem that existing style object insertion methods cannot seamlessly insert a specified object in a style image background.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows: A feature injection-based style object insertion method is provided, comprising the following steps: S1, obtaining a style background image , an object image to be inserted , and a mask corresponding to respectively an insertion position and an object region to be inserted and , preprocessing and inputting into a local redraw type diffusion model to generate an initial synthesis image ; S2, performing noise inversion processing on and respectively to obtain corresponding content noise and style noise, and extracting Q feature matrix of and KV feature matrix of from the self-attention layer during the inversion process; S3, mixing the content noise and the style noise to obtain mixed noise, taking the mixed noise as the starting point, and injecting the Q feature matrix and the KV feature matrix into the self-attention layer to perform guided denoising on , to obtain a synthesis image consistent with style .

[0006] Further, the preprocessing method of is as follows: S11, converting and to RGB format, binarizing and , and adjusting the size of to be consistent with the size of ; S12, based on , cutting the object to be inserted from and filling it with a white background, and based on the rectangular bounding box of , cropping and isometrically filling the filled image into a square to obtain a standardized object image ; S13, using the rectangular bounding box of to convert to a square and scale out, shielding the mask region of to obtain a shielding image , horizontally splicing and to obtain a double-sided panel image ; S14, constructing a double-sided panel mask of ​ Its expression is: in, To and Consistent size all-zero mask; This is for splicing operations.

[0007] Furthermore, the generation of locally redrawn diffusion models The methods include: S101, will Extract semantic prior features from the input Redux prior model; S102, according to reserve The unedited portion forms the input image. ,Will and The latent representation of the latent space is obtained by inputting it into the VAE encoder, and the latent representation is concatenated with the randomly initialized noise tensor to form the model input. S103. Semantic prior features and text prompts for guiding image generation are injected into the local redrawing diffusion model. The local redrawing diffusion model extracts features from the model input and text prompts using the DIT image branch and DIT text branch, respectively. All features are then fused and attention is calculated using a multi-head attention mechanism. After diffusion denoising, the model is generated. .

[0008] Furthermore, the attention calculation expression for the multi-head attention mechanism in step S103 is as follows: in, This represents a multi-head attention mechanism; Q, K, and V are the query vector, key vector, and value vector of the multi-head attention mechanism, respectively. , and They are respectively The latent representation includes the query vector, key vector, and value vector; , and They are respectively The query vector, key vector, and value vector; T is the transpose; is the normalized exponential function; d is the number of channels.

[0009] Furthermore, step S2 specifically includes: processing the data using the DDIM inversion method. and Perform noise inversion processing separately to obtain the content noise with a noise addition step count of T steps. and style noise ; and extract from the self-attention layer at each time step t during the inversion process. Q-feature matrix and The KV feature matrix.

[0010] Furthermore, the expression for mixing content noise and style noise is as follows: in, It is mixed noise; and These are the mean and standard deviation of the corresponding inputs, respectively.

[0011] Furthermore, the local redrawing diffusion model adopts the InsertAnything model.

[0012] This invention discloses a style object insertion method based on feature injection, the beneficial effects of which are: 1. This invention first generates an initial composite image using a local redrawing diffusion model, and then performs stylization processing by combining noise inversion and feature injection techniques, thereby achieving seamless insertion of specified objects into a style background image. Moreover, this invention does not require model training for a specific style or object, which significantly improves the convenience and versatility of operation. By mixing content noise with style noise and injecting key-value features of the style background during the denoising process, the style features of the background image can be effectively transferred to the inserted object, ensuring a high degree of consistency in visual style of the composite image.

[0013] 2. This invention provides a well-structured and clearly defined input for subsequent diffusion model processing by standardizing the input image and mask, including object matting, bounding box cropping and filling, and double-panel image stitching. The preprocessing method clarifies the spatial relationship between the reference object and the target editing area, providing precise positioning guidance for the model and helping to generate an initial composite image with accurate position and harmonious proportions, laying the foundation for subsequent high-quality style transfer.

[0014] 3. This invention introduces the Redux prior model to extract semantic prior features and combines dual-panel image input with VAE encoding, enabling the local redraw diffusion model to fully understand the identity details of the object to be inserted and the contextual information of the target insertion position; and utilizes the dual-branch structure of the DIT model to collaboratively process visual and textual information, and achieves accurate local redrawing through feature fusion, which can accurately generate the initial insertion result in complex style backgrounds and keep the semantic features of the object itself intact.

[0015] 4. This invention clarifies the specific composition of the query vector, key vector, and value vector in the multi-head attention mechanism, and integrates mask information and image content information into the attention calculation through channel concatenation. This design enables the model to not only focus on image content during generation but also accurately identify editable regions, thereby strictly limiting object generation to the specified mask area. This effectively avoids unnecessary modifications to non-editable background areas and improves the control precision of the generation process.

[0016] 5. By employing the DDIM inversion processing method, this invention can invert the initial composite image and the style background image to the noise space respectively, providing a basis for subsequent style noise mixing. During the inversion process, the feature matrix of the attention layer is extracted simultaneously, providing key guiding information for accurate style feature injection in the denoising stage, ensuring that the style transfer process is based on evidence and is carried out in a directional manner.

[0017] 6. This invention achieves effective fusion of content and style at the noise level by normalizing and adjusting the content noise using the statistical characteristics (mean and standard deviation) of style noise. This AdaIN-based hybridization method can preserve the content structure of the initial composite image while incorporating the global texture and color distribution features of the style background image, providing an optimized noise starting point for the final generation of a style-consistent image.

[0018] 7. This invention employs a dedicated InsertAnything model as a local redrawing diffusion model. This model has excellent performance in object insertion tasks and can directly utilize its powerful object understanding and context fusion capabilities, providing a reliable and efficient technical foundation for the initial composite graph generation step of this invention, thus ensuring the stability and superior performance of the overall process. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of a style object insertion method based on feature injection. Detailed Implementation

[0020] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0021] Considering that existing style object insertion methods cannot seamlessly insert specified objects into the background of styled images, such as in *AnyDoor: Zero-shot Object-level Image Customization*, which primarily injects style within the object and lacks co-modeling of the background, it fails to effectively transfer background style elements to the inserted object, resulting in style mismatch and poor integration, and thus cannot blend the inserted object well with the background. To address this issue, [reference...] Figure 1 This embodiment provides a style object insertion method based on feature injection, including the following steps: S1. Obtain the style background image Image of the object to be inserted and their respective corresponding Insertion position and Mask of the region to be inserted and ,Will Preprocessing is performed and the data is input into a locally redrawn diffusion model to generate an initial composite image. .

[0022] S2, to and Noise inversion is performed separately to obtain the corresponding content noise and style noise, and these are extracted from the self-attention layer during the inversion process. Q-feature matrix and The KV feature matrix.

[0023] S3. Mix content noise and style noise to obtain mixed noise. Using the mixed noise as a starting point, inject the Q-feature matrix and KV-feature matrix into the self-attention layer. Perform guided denoising to obtain and Composite images with consistent style .

[0024] Specifically, preprocessing The method is as follows: S11, will and Convert to RGB format, for and Perform binarization processing, and Adjust the size to match The dimensions are consistent.

[0025] Specifically, and Pixels with a value greater than 128 are set to 1 (foreground) and the rest to 0 (background), and the array is converted to uint8 type using Python array operations.

[0026] S12, based on from Extract the object to be inserted from the middle and fill it with a white background, and based on... The rectangular bounding box is used to crop the filled image and fill it proportionally to a square to obtain a standardized object image. .

[0027] S13. Use The rectangular bounding box will Transform it into a square and expand it proportionally. The masked area is used to block the image to obtain the shielded image. ,Will and Horizontal stitching is performed to obtain a double-sided image. .

[0028] S14, Construction double-sided mask Its expression is: in, To and Consistent size all-zero mask; This is for splicing operations.

[0029] By performing standardized preprocessing on the input image and mask, including object matting, bounding box cropping and filling, and double-panel image stitching, a well-structured and clearly defined input is provided for subsequent diffusion model processing. The preprocessing method clarifies the spatial relationship between the reference object and the target editing area, providing precise positioning guidance for the model and helping to generate an initial composite image with accurate position and harmonious proportions, laying the foundation for subsequent high-quality style transfer.

[0030] In this embodiment, the local redrawing diffusion model uses the InsertAnything model from "Insert Anything: Image Insertion via In-Context Editing in DiT" and injects the LoRA weights, Flux Fill weights, and Flux Redux weights of InsertAnything. The local redrawing diffusion model is generated. The methods include: S101, will Extract semantic prior features from the input Redux prior model.

[0031] S102, according to reserve The unedited portion forms the input image. ,Will and The latent representation of the latent space is obtained by inputting it into the VAE encoder, and the latent representation is concatenated with the randomly initialized noise tensor to form the model input.

[0032] S103. In the local redrawing diffusion model, semantic prior features and text prompts for guiding image generation are injected. The local redrawing diffusion model extracts features from the model input and text prompts using the DIT image branch and DIT text branch respectively, and fuses all features through a multi-head attention mechanism, performs attention calculation, and generates the final image after diffusion denoising. .

[0033] Furthermore, the attention calculation expression for the multi-head attention mechanism in step S103 is as follows: in, This represents a multi-head attention mechanism; Q, K, and V are the query vector, key vector, and value vector of the multi-head attention mechanism, respectively. , and They are respectively The latent representation includes the query vector, key vector, and value vector; , and They are respectively The query vector, key vector, and value vector; T is the transpose; d is the normalized exponential function; d is the number of channels. By clarifying the specific composition of the query vector, key vector, and value vector in the multi-head attention mechanism, mask information and image content information are concatenated through channels to participate in attention calculation. This design enables the model to not only focus on image content during generation but also accurately identify editable regions, thus strictly limiting object generation to the specified mask range. This effectively avoids unnecessary modifications to non-editable background areas and improves the control precision of the generation process.

[0034] Step S2 specifically includes: processing the data using the DDIM inversion method. and Perform noise inversion processing separately to obtain the content noise with a noise addition step count of T steps. and style noise ; and extract from the self-attention layer at each time step t during the inversion process. Q-feature matrix and The KV feature matrix is ​​obtained. By adopting the DDIM inversion processing method, the initial synthetic image and the style background image can be inverted to the noise space respectively, providing a basis for subsequent style noise mixing. During the inversion process, the feature matrix from the attention layer is extracted simultaneously, providing key guiding information for accurate style feature injection in the denoising stage, ensuring that the style transfer process is based on evidence and is carried out in a targeted manner.

[0035] Specifically, the expression for mixing content noise and style noise is as follows: in, It is mixed noise; and These represent the mean and standard deviation of the corresponding inputs, respectively. The AdaIN-based blending method preserves the content structure of the initial synthesized image while incorporating the global texture and color distribution features of the style background image, providing an optimized noise starting point for generating a style-consistent image in the end.

[0036] In summary, the beneficial effects of this solution are as follows: This solution first uses a local redrawing diffusion model to generate an initial composite image, and then combines noise inversion and feature injection techniques for stylization processing, thereby enabling the seamless insertion of specified objects into a stylized background image.

[0037] Meanwhile, this solution does not require model training for specific styles or objects, which significantly improves the ease of operation and versatility. By mixing content noise with style noise and injecting key features of style background during the denoising process, the style features of the background image can be effectively transferred to the inserted object, ensuring a high degree of consistency in visual style of the synthesized image.

[0038] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this invention.

Claims

1. A style object insertion method based on feature injection, characterized in that, Includes the following steps: S1. Obtain the style background image Image of the object to be inserted and their respective corresponding Insertion position and Mask of the area to be inserted and ,Will Preprocessing is performed and the data is input into a locally redrawn diffusion model to generate an initial composite image. ; S2, to and Noise inversion is performed separately to obtain the corresponding content noise and style noise, and these are extracted from the self-attention layer during the inversion process. Q-feature matrix and The KV feature matrix; S3. Mix content noise and style noise to obtain mixed noise. Using the mixed noise as a starting point, inject the Q-feature matrix and KV-feature matrix into the self-attention layer. Perform guided denoising to obtain and Composite images with consistent style .

2. The style object insertion method based on feature injection according to claim 1, characterized in that, Preprocessing The method is as follows: S11, will and Convert to RGB format, for and Perform binarization processing, and Adjust the size to match Consistent dimensions; S12, based on from Extract the object to be inserted from the middle and fill it with a white background, and based on... The rectangular bounding box is used to crop the filled image and fill it proportionally to a square to obtain a standardized object image. ; S13. Use The rectangular bounding box will Transform it into a square and expand it proportionally. The masked area is used to block the image to obtain the shielded image. ,Will and Horizontal stitching is performed to obtain a double-sided image. ; S14, Construction double-sided mask Its expression is: in, To and Consistent size all-zero mask; This is for splicing operations.

3. The style object insertion method based on feature injection according to claim 2, characterized in that, Local redrawing diffusion model generation The methods include: S101, will Extract semantic prior features from the input Redux prior model; S102, according to reserve The unedited portion forms the input image. ,Will and The latent representation of the latent space is obtained by inputting it into the VAE encoder, and the latent representation is concatenated with the randomly initialized noise tensor to form the model input. S103. Semantic prior features and text prompts for guiding image generation are injected into the local redrawing diffusion model. The local redrawing diffusion model extracts features from the model input and text prompts using the DIT image branch and DIT text branch, respectively. All features are then fused and attention is calculated using a multi-head attention mechanism. After diffusion denoising, the model is generated. .

4. The style object insertion method based on feature injection according to claim 3, characterized in that, The attention calculation expression for the multi-head attention mechanism in step S103 is as follows: in, This represents a multi-head attention mechanism; Q, K, and V are the query vector, key vector, and value vector of the multi-head attention mechanism, respectively. , and They are respectively The latent representation includes the query vector, key vector, and value vector; , and They are respectively The query vector, key vector, and value vector; T is the transpose; is the normalized exponential function; d is the number of channels.

5. The style object insertion method based on feature injection according to claim 3, characterized in that, Step S2 specifically includes: processing the data using the DDIM inversion method. and Perform noise inversion processing separately to obtain the content noise with a noise addition step count of T steps. and style noise ; and extract from the self-attention layer at each time step t during the inversion process. Q-feature matrix and The KV feature matrix.

6. The style object insertion method based on feature injection according to claim 3, characterized in that, The expression for mixing content noise and style noise is: in, It is mixed noise; and These are the mean and standard deviation of the corresponding inputs, respectively.

7. The style object insertion method based on feature injection according to claim 3, characterized in that, The local redrawing diffusion model adopts the InsertAnything model.

Images