Image filling method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of image processing, and provides an image filling method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: encoding a text prompt word to obtain a text encoding vector, and encoding a masked background image to obtain an image encoding vector; splicing the image encoding vector, a position mask and random noise to obtain a fusion noise vector; gradually denoising the fusion noise vector, and adding the text encoding vector and a texture guide vector as guide conditions in the denoising process to generate a filling image; and the texture guide vector is obtained based on splicing, smooth transition and encoding processing of texture features of a foreground image and texture features of a background image. Through the gradual denoising process and in combination with the text encoding vector and the texture guide vector as double guide conditions, the filling image with rich details, natural texture and high fusion with the background image can be generated.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image filling method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image inpainting, a key technology in image processing, aims to restore or repair occluded or missing parts of an image, striving to achieve a visually natural and complete effect. This technology has a wide range of applications and significant commercial value, such as image restoration, background replacement, virtual try-on, and industrial defect synthesis.

[0003] Traditional image inpainting methods primarily rely on information such as color and texture surrounding the target region, using diffusion mechanisms or sample-based methods to fill in missing areas. However, these methods fall short when dealing with complex scenes. For example, diffusion-based methods often produce artifacts and blurring when handling large or textured missing areas, resulting in unsatisfactory restoration results. Sample-based methods, on the other hand, are limited by computational efficiency and advanced feature extraction capabilities, making it difficult to achieve high-quality image restoration while maintaining restoration speed.

[0004] With the rapid development of artificial intelligence (AI) technology, especially the rise of AI-generated content (AICG), deep learning-based image inpainting methods have emerged. These methods utilize advanced techniques such as convolutional neural networks (CNNs) and generative adversarial networks (GANs) to capture the semantic features of image context and generate images. However, despite these methods' progress in image inpainting, they still face numerous challenges. For example, balancing detailed features with overall structure, achieving high-quality image inpainting in complex scenes, and ensuring that the foreground of the inpainted image maintains high consistency with its surroundings or that the foreground and background transition naturally are all problems that urgently need to be solved. Summary of the Invention

[0005] This invention provides an image filling method, apparatus, electronic device, and storage medium to solve the defects in related technologies where the filled image has poor foreground consistency or unnatural foreground-background transition, resulting in unsatisfactory image filling effects.

[0006] This invention provides an image filling method, comprising:

[0007] The text prompts are encoded to obtain a text encoding vector, and the masked background image is encoded to obtain an image encoding vector. The text prompts are used to describe the content of the generated image.

[0008] The image encoding vector, the position mask, and random noise are concatenated to obtain a fused noise vector. The position mask is used to indicate the filling position of the foreground image in the background image.

[0009] The fused noise vector is progressively denoised, and the text encoding vector and texture guidance vector are added as guidance conditions during the denoising process to generate a filled image. The texture guidance vector is obtained by splicing, smoothing transition and encoding the texture features of the foreground image and the background image.

[0010] According to an image filling method provided by the present invention, the step of obtaining the texture guiding vector includes:

[0011] Texture features are extracted from the foreground image and the background image respectively to obtain foreground texture features and background texture features. The foreground texture features and the background texture features are then concatenated to obtain concatenated texture features.

[0012] The splicing positions of the spliced ​​texture features are smoothed to obtain smooth texture features, and the smooth texture features are encoded to obtain the texture guiding vector.

[0013] According to an image filling method provided by the present invention, the step of concatenating the foreground texture features and the background texture features to obtain concatenated texture features includes:

[0014] Based on the location mask, the background texture features are masked to obtain the masked background texture features;

[0015] Based on the mask size, the size of the foreground texture feature is adjusted to obtain the adjusted foreground texture feature;

[0016] The adjusted foreground texture features and the masked background texture features are concatenated to obtain the concatenated texture features.

[0017] According to an image filling method provided by the present invention, the step of smoothing the transition at the splicing position of the spliced ​​texture features to obtain smooth texture features includes:

[0018] The target number of pixels is expanded both inward and outward from the splicing position of the spliced ​​texture feature to obtain the ring-shaped region to be processed. The target number is determined based on the diffusion time step.

[0019] Based on the diffusion time step, the annular region to be processed is subjected to Gaussian blurring to obtain the smooth transition texture feature.

[0020] According to an image filling method provided by the present invention, the number of targets decreases as the diffusion time step decreases, and the degree of Gaussian blurring of the annular region to be processed decreases as the diffusion time step decreases.

[0021] According to an image filling method provided by the present invention, the step of concatenating the image encoding vector, the position mask, and random noise to obtain a fused noise vector includes:

[0022] The location mask is downsampled to obtain the downsampled mask;

[0023] The image encoding vector, the downsampled mask, and the random noise are concatenated to obtain a fused noise vector, wherein the image encoding vector, the downsampled mask, and the random noise have the same size.

[0024] According to an image filling method provided by the present invention, the step of progressively denoising the fused noise vector and adding the text encoding vector and texture guiding vector as guiding conditions during the denoising process to generate a filled image includes:

[0025] Based on the diffusion model, the text encoding vector and the texture guidance vector are applied to progressively denoise the fused noise vector to obtain a denoised feature vector. The text encoding vector is used as a text guidance condition input to the attention module of the encoder and decoder of the diffusion model, and the texture guidance vector is used as an image guidance condition input to the attention module of the encoder of the diffusion model.

[0026] The denoised feature vectors are decoded to obtain the filled image.

[0027] The present invention also provides an image filling device, comprising:

[0028] The encoding unit is used to encode the text prompt words to obtain a text encoding vector, and to encode the masked background image to obtain an image encoding vector. The text prompt words are used to describe the generated image content.

[0029] The splicing unit is used to splice the image encoding vector, the position mask, and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image;

[0030] The generation unit is used to progressively denoise the fused noise vector and add the text encoding vector and texture guidance vector as guidance conditions during the denoising process to generate a filled image. The texture guidance vector is obtained by splicing, smoothing transition and encoding the texture features of the foreground image and the texture features of the background image.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the image filling method as described above.

[0032] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image filling method as described above.

[0033] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image filling method as described above.

[0034] The image filling method, apparatus, electronic device, and storage medium provided by this invention generate filled images that are rich in detail, have natural textures, and are highly integrated with the background image by progressively denoising the fused noise vector and combining text encoding vectors and texture guidance vectors as dual guidance conditions during the denoising process. The introduction of texture guidance vectors allows the texture features of the foreground image to smoothly transition with the background image, further enhancing the realism and visual appeal of the filled image. Furthermore, by encoding text prompts, the user's text description can be directly converted into specific guidance for image filling, increasing the flexibility and personalization of image generation. By using positional masks to indicate the specific filling position of the foreground image in the background image, the image filling process becomes more precise and controllable, thereby avoiding problems such as image misalignment or inaccurate filling areas and ensuring the visual consistency and coherence between the foreground and background of the generated filled image. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the image filling method provided by the present invention;

[0037] Figure 2 This is a schematic diagram of the annular region to be processed provided by the present invention;

[0038] Figure 3 This is a flowchart of the texture-guided image filling method provided by the present invention;

[0039] Figure 4This is a schematic diagram of the image filling device provided by the present invention;

[0040] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] Image inpainting is an image processing technique designed to restore or repair occluded or missing parts of an image, making them appear as natural and complete as possible. Traditional image inpainting techniques fill in the target area based on information such as color and texture around the target area, using diffusion mechanisms or sample-based methods. In recent years, artificial intelligence technology, especially AIGC technology, has made rapid progress, with new deep learning-based generative methods emerging one after another. Users' needs for image inpainting are no longer limited to repairing and filling in based on image features around the target area. A new application scenario has emerged: filling missing areas or specified target areas of a background image with user-specified foreground image content, ensuring that the filled foreground image maintains consistency with the background image, while maintaining a high degree of harmony in the transition between the foreground and background images.

[0043] Currently, image inpainting methods are mainly divided into two categories: traditional methods and generative methods based on deep learning. Traditional inpainting methods can be further subdivided into diffusion-based methods, sample-based methods, and Poisson fusion methods, among others. Deep learning-based image inpainting methods are mainly divided into convolutional neural network-based methods, generative adversarial network-based methods, and diffusion model-based methods.

[0044] However, the above methods all have certain drawbacks in practical applications. Diffusion-based methods primarily utilize the edge information of the region to be repaired to determine the diffusion direction, then smoothly diffuse inwards from the edge to fill in known information. This type of method achieves remarkable image restoration results when the damaged area is small, but when the missing area becomes larger or the texture is complex, the restoration result will produce artifact-like blurring. Furthermore, this type of method also suffers from inconsistencies in structure and texture, and unreasonable restoration content. Sample-based methods calculate the similarity between the missing area and known areas, obtain the sample with the highest similarity, and copy it into the missing area to achieve image restoration. This type of method has excessively long computation time and difficulty in extracting high-level image features, resulting in less than ideal image restoration results and difficulty in ensuring the reasonableness of the restored image. Poisson fusion-based image fusion methods utilize the Poisson equation in mathematics to achieve a smooth transition and seamless fusion of foreground and background images. However, in some cases, Poisson fusion may affect the original color of the target image, leading to visual discontinuity between the fused area and the surrounding environment.

[0045] Image inpainting methods based on convolutional neural networks (CNNs) use CNNs as feature extraction methods, capturing semantic features through convolution operations. These methods mainly consist of two steps: encoding and decoding. The encoder captures the contextual semantic features of the image, while the decoder uses these features to generate the missing image content. Image inpainting methods based on generative adversarial networks (GANs) mainly consist of a generator and a discriminator. The generator aims to produce an image, while the discriminator aims to distinguish between the generated and real images. These methods gradually generate realistic images through continuous interaction between the generator and the discriminator, thus achieving the goal of image restoration. Diffusion models are generative models based on probability density functions. This technique combines natural language processing and computer vision, realizing the transformation from text description to image generation. It generates data by gradually diffusing random noise in a latent space. Compared to other generative models, diffusion models have better generation quality and interpretability, thus showing broad application prospects in the field of image generation.

[0046] However, current methods for image filling using diffusion models often only use either text guidance or image guidance, ignoring the complementarity of the two types of information. Using text guidance alone lacks control over the content generated by the diffusion model, while using image guidance alone ignores the large-scale semantic information utilized during model training, resulting in poor foreground consistency or unnatural foreground-background transitions after filling.

[0047] To address this issue, this invention proposes a texture-guided image infilling method that combines the interpretability advantages of traditional methods with the high-quality performance guarantee of diffusion models. Based on a pre-trained diffusion model, the Gabor operator is used to extract texture features from both the background and foreground images. While retaining the high-order semantic information provided by text guidance, a proposed texture transition module incorporates the texture features of the foreground and background stitching as additional guidance conditions for the diffusion model. This enables the diffusion model to complete the image infilling task more accurately and controllably, thereby overcoming the aforementioned shortcomings.

[0048] It should be noted that the overall input of the method proposed in this invention includes three parts: a text prompt, a background image, a location mask for generating the foreground, and a foreground image. The text prompt describes the content of the final generated image. Structurally, the method proposed in this invention can be divided into two modules: a pre-trained diffusion model and a texture guidance module. The pre-trained diffusion model includes a text encoder, an image encoder, a denoising model U-Net, and an image decoder. The texture guidance module includes a Gabor filter, a texture transition module, and an image encoder. The image filling method provided by this invention will be described in detail below in conjunction with the system modules.

[0049] Figure 1 This is a flowchart illustrating the image filling method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0050] Step 110: Encode the text prompt words to obtain a text encoding vector, and encode the masked background image to obtain an image encoding vector. The text prompt words are used to describe the generated image content.

[0051] Specifically, text prompts are descriptive texts used to indicate or describe the content of the final generated image. For example, in an image fill task, a text prompt might be "a cat on the floor." This textual information provides semantic guidance for the image fill process, ensuring that the generated filled image meets or closely approximates the user's expectations.

[0052] The text encoder in the pre-trained diffusion model can encode the input text prompts to obtain a text encoding vector. Here, the text encoding vector refers to the vector representation obtained by encoding the text prompts. This vector representation can capture the semantic information in the text and serve as a guiding condition (i.e., text guiding condition) in the image filling process.

[0053] The masked background image refers to the image obtained by applying a positional mask to the original background image. A positional mask is a binary (or grayscale) image of the same size as the background image, used to indicate the filling positions of the foreground image within the background image. It defines which areas are the foreground (the areas to be filled) and which areas are the background (the areas to be preserved). By applying this mask to the background image, some areas of the background image can be preserved, while other areas (i.e., the foreground areas) are set to 0 or a specific value for subsequent image filling.

[0054] The image encoder in the pre-trained diffusion model can encode the masked background image to obtain an image encoding vector. Here, the image encoding vector refers to the vector representation obtained by encoding the masked background image. This vector representation can capture the visual features in the image, such as color, texture, and shape.

[0055] Step 120: The image encoding vector, the position mask, and the random noise are concatenated to obtain a fused noise vector. The position mask is used to indicate the filling position of the foreground image in the background image.

[0056] Specifically, after obtaining the image encoding vector, the image encoding vector, the position mask, and random noise of the same size can be concatenated to obtain a fused noise vector, which is then used as input to the denoising model U-Net. Here, the fused noise vector is the result of the above concatenation operation, containing all the information from the image encoding vector, the position mask, and the random noise. This vector will serve as input to the subsequent denoising process, gradually denoising and incorporating text encoding vectors and texture guidance vectors as guiding conditions to ultimately generate a filled image.

[0057] Understandably, this step, by concatenating the image encoding vector, the location mask, and random noise, aims to generate a fused noise vector. This vector contains image information, location information, and randomness information. The introduction of random noise is intended to increase the diversity of the generated images and avoid generating overly deterministic or repetitive images. By concatenating this information, the model can simultaneously consider image content, location information, and randomness when generating images, thereby producing images that better meet expectations.

[0058] Step 130: The fused noise vector is gradually denoised, and the text encoding vector and texture guidance vector are added as guidance conditions during the denoising process to generate a filled image. The texture guidance vector is obtained by splicing, smoothing transition and encoding the texture features of the foreground image and the background image.

[0059] Specifically, progressive denoising refers to gradually reducing or eliminating noise or unnecessary details in an image through a series of steps or iterations during the generation of a filled image, thereby obtaining a clearer, smoother, or more desirable image. In the denoising process, adding text encoding vectors and texture guidance vectors provides the U-Net denoising model with additional semantic and visual information, guiding the model to generate images that better meet expectations.

[0060] The text-encoded vector, serving as a text-guided condition, contains descriptive information about the content of the generated image. By incorporating this information into the denoising process, it ensures that the generated image matches the content described by the text prompts. The texture-guided vector, serving as an image-guided condition, is obtained by concatenating, smoothing, and encoding the texture features of the foreground and background images. This vector captures texture information in the image, guiding the generation of image regions with similar textures. It should be understood that the texture-guided vector is a special vector representation that combines the texture features of both the foreground and background images. By concatenating, smoothing, and encoding these features, a smooth vector containing both foreground and background texture information is obtained. This vector is used as a guide condition during the denoising process to ensure that the generated image maintains textural consistency with the foreground and background images.

[0061] Specifically, when progressively denoising the fused noise vector, the fused noise vector can be used as input to the denoising model U-Net. Through a series of iterative operations, the denoising model gradually reduces or eliminates noise in the image. This involves operations such as image feature extraction, transformation, and reconstruction. In each denoising process, text encoding vectors and texture guidance vectors are added as guiding conditions. These vectors are combined with the current denoising result in a specific way to guide the generation of an image that better meets the desired result. For example, the text encoding vector can be used as the text guiding condition, input into the attention module of the encoder / decoder of the denoising model U-Net as the K value, while the texture guidance vector can be used as the image guiding condition, input into the attention module of the encoder of the denoising model U-Net as the V value. After multiple iterations, the denoising model outputs the final features after T-step diffusion (i.e., the denoising process), and the filled image can be obtained using the image decoder.

[0062] Here, the filled image refers to the image obtained through progressive denoising and the addition of guiding conditions. It combines the contextual information of the background image, the texture features of the foreground image, and the semantic information described by the text prompts. The generated filled image is visually consistent with the original image and conforms to the description of the text prompts in terms of content.

[0063] The method provided in this invention generates a filled image that is rich in detail, has a natural texture, and blends seamlessly with the background image by progressively denoising the fused noise vector and combining text encoding vectors and texture guidance vectors as dual guiding conditions during the denoising process. The introduction of texture guidance vectors allows for a smooth transition between the texture features of the foreground image and the background image, further enhancing the realism and visual appeal of the filled image. Furthermore, by encoding text prompts, the user's text description can be directly converted into specific guidance for image filling, increasing the flexibility and personalization of image generation. By using positional masks to indicate the specific filling position of the foreground image in the background image, the image filling process becomes more precise and controllable, thereby avoiding problems such as image misalignment or inaccurate filling areas and ensuring visual consistency and coherence between the foreground and background of the generated filled image.

[0064] Based on the above embodiments, the step of obtaining the texture guidance vector includes:

[0065] Step 210: Extract texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and then concatenate the foreground texture features and the background texture features to obtain concatenated texture features.

[0066] Specifically, the texture guidance module processes the foreground and background images accordingly to obtain the texture guidance vector. Specifically, the foreground and background images are each individually processed by a Gabor filter to extract multi-directional texture features as prior information. This Gabor filter can be a 2D Gabor filter. The Gabor kernel is used to filter the image, describing the local frequency information of the signal. The kernel function of the Gabor filter can be expressed as:

[0067]

[0068] in:

[0069]

[0070]

[0071] and The pixel position of the Gabor filter kernel. These represent the wavelength, angle, phase offset, standard deviation, and ellipticity of the filter core, respectively.

[0072] Further, in step 210, the step of concatenating the foreground texture feature and the background texture feature to obtain the concatenated texture feature includes:

[0073] Step 211: Based on the position mask, perform masking processing on the background texture features to obtain the masked background texture features;

[0074] Step 212: Based on the mask size, adjust the size of the foreground texture feature to obtain the adjusted foreground texture feature;

[0075] Step 213: The adjusted foreground texture feature and the masked background texture feature are spliced ​​together to obtain the spliced ​​texture feature.

[0076] Specifically, after extracting background and foreground texture features using a Gabor filter, preprocessing is required before concatenating these features. First, the extracted texture features from the background image are multiplied by a positional mask to obtain masked background texture features. Then, the foreground texture features are resized according to the mask size (adjusting the texture feature size to meet specific requirements). This is because if the foreground and background texture features are not the same size, they cannot be directly concatenated. Therefore, the foreground texture features need to be adjusted to the same size as the mask (i.e., the masked portion of the background texture features) to facilitate subsequent concatenation. Finally, the resized foreground texture features are assigned to the masked background texture features to obtain the concatenated texture features.

[0077] Step 220: Perform smooth transition processing on the splicing position of the spliced ​​texture features to obtain smooth texture features, and encode the smooth texture features to obtain the texture guiding vector.

[0078] It should be noted that directly splicing the texture features of the foreground and background can easily result in an unnatural transition at the splicing position. Therefore, this embodiment of the invention designs a texture transition module related to the diffusion time step t to perform Gaussian blur at the splicing position, so that the texture features are smooth at the splicing position, and the texture connectivity restriction at the splicing position is gradually reduced as the diffusion time step t of the denoising model decreases.

[0079] Here, denoising models (also known as diffusion models) are a type of generative model that generates new data by progressively adding noise to the data and then learning a reverse process (i.e., progressively generating data from the noise). In denoising models, the diffusion time step t refers to a point in time in the progressive transformation from the original data to complete noise (or vice versa). In forward diffusion, t represents the number or degree of noise added; in reverse diffusion (i.e., generation), t represents the number or degree of steps to progressively recover the original data from the noise.

[0080] Further, in step 220, the smoothing transition processing of the splicing positions of the spliced ​​texture features to obtain smooth texture features includes:

[0081] Step 221: Expand the splicing position of the splicing texture feature by a target number of pixels both inside and outside to obtain the ring-shaped region to be processed. The target number is determined based on the diffusion time step.

[0082] Step 222: Based on the diffusion time step, perform Gaussian blur processing on the annular region to be processed to obtain the smooth transition texture feature.

[0083] Figure 2 This is a schematic diagram of the annular region to be processed provided by the present invention, as shown below. Figure 2 As shown, the specific process for achieving smooth texture transition is as follows: For time step t of the diffusion model, expand by d pixels both inside and outside the stitching position to obtain the annular region to be processed. Here, d is the target number, and its calculation formula is as follows:

[0084]

[0085] Where a and b are predefined parameters, d gradually decreases as time step t decreases, and the constraints of texture features used as image guides also gradually decrease. Within a 2d range of the image stitching position after foreground and background stitching, a Gaussian blur related to time step t is applied, and the Gaussian kernel is defined as:

[0086]

[0087] in and The pixel position is the Gaussian kernel. Let be the standard deviation of the Gaussian kernel. As the time step t decreases, the content generated by the denoising model gradually becomes clearer. Therefore, we want to reduce the degree of Gaussian blurring at the splicing position to enhance the texture guidance intensity. So, we define:

[0088]

[0089] Where c is a hyperparameter, the simplified Gaussian kernel is defined as follows:

[0090]

[0091] For each time step t, the stitched texture features are processed by the texture transition module to obtain smooth transition texture features. This feature is input into the image encoder, and the resulting vector is the texture guidance vector. This vector is used as the image guidance condition and input into the attention module of the encoder of the denoising model U-Net as the V value.

[0092] Based on any of the above embodiments, step 120 specifically includes:

[0093] Step 121: Downsample the location mask to obtain the downsampled mask;

[0094] Step 122: The image encoding vector, the downsampled mask, and the random noise are concatenated to obtain a fused noise vector, wherein the image encoding vector, the downsampled mask, and the random noise have the same size.

[0095] Specifically, before concatenating the fused noise vector, the location mask can be downsampled to better match its size with the image encoding vector, while reducing its resolution, removing redundant information, and decreasing the computational load of subsequent processing steps. Then, the image encoding vector, the downsampled mask, and random noise of the same size can be concatenated. Here, random noise, typically used to increase the diversity of the generated image or simulate uncertainty in the image, is adjusted to the same size as the encoded vector and mask so that they can be concatenated together.

[0096] Specifically, before concatenation, it's crucial to ensure that the image encoding vector, the downsampled mask, and the random noise have the same dimensions. If their dimensions differ, appropriate adjustments (such as padding, cropping, or adjusting the number of channels) can be made to match the dimensions. Subsequently, they can be concatenated along the channel dimensions to form a comprehensive input containing image encoding information, location mask information, and random noise information. This comprehensive input can then be used as input to the denoising model U-Net for further processing.

[0097] Based on any of the above embodiments, step 130 specifically includes:

[0098] Step 131: Based on the diffusion model, the text encoding vector and the texture guidance vector are applied to progressively denoise the fused noise vector to obtain a denoised feature vector. The text encoding vector is used as a text guidance condition input to the attention module of the encoder and decoder of the diffusion model, and the texture guidance vector is used as an image guidance condition input to the attention module of the encoder of the diffusion model.

[0099] Step 132: Decode the denoised feature vector to obtain the filled image.

[0100] Specifically, after concatenating the fused noise vector, this vector is used as the input to the denoising model U-Net. At different time steps t, the U-Net encoder acquires foreground and background texture features processed to varying degrees after concatenation. These features serve as strong prior information to guide the output of the diffusion model at each step, enhancing the consistency of the diffusion model in preserving the specified foreground in the image filling task. The final features after the T-step diffusion process of the denoising model are then decoded using an image decoder to obtain the filled image.

[0101] Based on any of the above embodiments Figure 3 This is a flowchart of the texture-guided image filling method provided by the present invention, such as... Figure 3 As shown, the overall input of this method can be divided into three parts: (1) a text prompt word to describe the content of the final generated image; (2) a background image and a position mask used to generate the foreground; and (3) a foreground image.

[0102] The method proposed in this embodiment can be divided into two modules in terms of structure: (1) a pre-trained diffusion model, including a text encoder, an image encoder, a denoising model U-Net, and a final image decoder; (2) a texture guidance module, including a Gabor filter, a foreground and background texture transition module, and an image encoder.

[0103] For the texture guidance module, the foreground and background original images are each separately processed by Gabor filtering to extract multi-directional texture features as prior information. Then, the texture features extracted from the background are multiplied by a mask to obtain the masked background texture features. Next, the texture features of the foreground are resized according to the mask size. Subsequently, the resized foreground texture features can be assigned to the masked background texture features to obtain the stitched texture features.

[0104] Here, directly stitching together the texture features of the foreground and background can easily result in unnatural transitions at the stitching position. Therefore, this embodiment of the invention designs a texture transition module related to the diffusion time step t, specifically tailored to the characteristics of the U-Net denoising model. This module performs Gaussian blurring at the stitching position to smooth the texture features at the stitching location and gradually reduces the texture connectivity constraints at the stitching point as the diffusion process t decreases. Specifically, for the time step t of the denoising model, the area to be processed is expanded by d pixels both inwards and outwards at the stitching position. As the time step t decreases, d gradually decreases, and the constraints on the texture features used for image guidance also gradually decrease. Within a 2d range (i.e., the area to be processed) of the stitching position after foreground and background stitching, Gaussian blurring related to the time step t is performed. For each time step t, the stitched texture features are processed by the texture transition module and then input into the image encoder. The resulting vector is used as the image guidance condition and input into the attention module of the encoder of the U-Net denoising model as the V value. It should be noted that... Figure 3 The intensity superposition strategy in this context refers to using the diffusion time step t to perform Gaussian blurring on the splicing position. As the time step t decreases, the degree of Gaussian blurring also decreases, thereby enhancing the intensity of texture guidance.

[0105] For the pre-trained diffusion model, the text encoder encodes the input text prompts, and the encoded text vector serves as the text-guided conditional input to the attention module of the U-Net encoder / decoder as the K value. The image encoder encodes the masked background image, and the encoded vector, the downsampled mask, and random noise of the same size are concatenated as input to the U-Net denoising model. At different time steps t, the U-Net encoder captures the concatenated foreground and background texture features with varying degrees of processing, serving as strong prior information to guide the output of the denoising model at each step, enhancing the consistency of the denoising model in preserving the specified foreground in the image filling task. The final features after the T-step diffusion process of the denoising model are decoded using the image decoder to obtain the filled image. It should be understood that... Figure 3 Z in T Z represents the input of the denoising model U-Net at time step T. T-1 Z0 represents the output of the denoising model U-Net after the first denoising process, and Z0 is the output of the denoising model after time step T (i.e. after T denoising processes).

[0106] The method provided in this invention uses a pre-trained diffusion model and the Gabor operator to extract texture features from the background and foreground images. While retaining the high-order semantic information provided by the text guidance, the proposed texture transition module uses the texture features of the two images as additional diffusion model guidance conditions, enabling the diffusion model to complete the image filling task more accurately and controllably.

[0107] Based on any of the above embodiments Figure 4 This is a schematic diagram of the image filling device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0108] The encoding unit 410 is used to encode the text prompt words to obtain a text encoding vector, and to encode the masked background image to obtain an image encoding vector. The text prompt words are used to describe the generated image content.

[0109] The splicing unit 420 is used to splice the image encoding vector, the position mask and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image;

[0110] The generation unit 430 is used to perform stepwise denoising on the fused noise vector, and to add the text encoding vector and texture guidance vector as guidance conditions during the denoising process to generate a filled image. The texture guidance vector is obtained by splicing, smoothing transition and encoding the texture features of the foreground image and the texture features of the background image.

[0111] The apparatus provided in this invention generates a filled image that is rich in detail, has a natural texture, and is highly integrated with the background image by progressively denoising the fused noise vector and combining text encoding vectors and texture guidance vectors as dual guidance conditions during the denoising process. The introduction of texture guidance vectors allows the texture features of the foreground image to smoothly transition with the background image, further enhancing the realism and visual appeal of the filled image. Furthermore, by encoding text prompts, the user's text description can be directly converted into specific instructions for image filling, increasing the flexibility and personalization of image generation. By using positional masks to indicate the specific filling position of the foreground image in the background image, the image filling process becomes more precise and controllable, thereby avoiding problems such as image misalignment or inaccurate filling areas and ensuring the visual consistency and coherence between the foreground and background of the generated filled image.

[0112] Based on any of the above embodiments, the device further includes a texture guiding unit, the texture guiding unit comprising:

[0113] The feature extraction unit is used to extract texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and to concatenate the foreground texture features and the background texture features to obtain concatenated texture features;

[0114] The texture transition unit is used to perform smooth transition processing on the splicing position of the spliced ​​texture features to obtain smooth texture features, and to encode the smooth texture features to obtain the texture guiding vector.

[0115] Based on any of the above embodiments, the feature extraction unit includes a feature splicing subunit, which is used for:

[0116] Based on the location mask, the background texture features are masked to obtain the masked background texture features;

[0117] Based on the mask size, the size of the foreground texture feature is adjusted to obtain the adjusted foreground texture feature;

[0118] The adjusted foreground texture features and the masked background texture features are concatenated to obtain the concatenated texture features.

[0119] Based on any of the above embodiments, the texture transition unit is specifically used for:

[0120] The target number of pixels is expanded both inward and outward from the splicing position of the spliced ​​texture feature to obtain the ring-shaped region to be processed. The target number is determined based on the diffusion time step.

[0121] Based on the diffusion time step, the annular region to be processed is subjected to Gaussian blurring to obtain the smooth transition texture feature.

[0122] Based on any of the above embodiments, the number of targets decreases as the diffusion time step decreases, and the degree of Gaussian blurring of the annular region to be processed decreases as the diffusion time step decreases.

[0123] Based on any of the above embodiments, the splicing unit 420 is specifically used for:

[0124] The location mask is downsampled to obtain the downsampled mask;

[0125] The image encoding vector, the downsampled mask, and the random noise are concatenated to obtain a fused noise vector, wherein the image encoding vector, the downsampled mask, and the random noise have the same size.

[0126] Based on any of the above embodiments, the generation unit 430 is specifically used for:

[0127] Based on the diffusion model, the text encoding vector and the texture guidance vector are applied to progressively denoise the fused noise vector to obtain a denoised feature vector. The text encoding vector is used as a text guidance condition input to the attention module of the encoder and decoder of the diffusion model, and the texture guidance vector is used as an image guidance condition input to the attention module of the encoder of the diffusion model.

[0128] The denoised feature vectors are decoded to obtain the filled image.

[0129] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an image filling method, which includes: encoding text prompts to obtain text encoding vectors, and encoding a masked background image to obtain an image encoding vector, wherein the text prompts are used to describe the content of the generated image; concatenating the image encoding vector, a position mask, and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; progressively denoising the fused noise vector, and adding the text encoding vector and a texture guidance vector as guidance conditions during the denoising process to generate a filled image, wherein the texture guidance vector is obtained by concatenating, smoothing transitions, and encoding the texture features of the foreground image and the background image.

[0130] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0131] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image filling method provided by the above methods. The method includes: encoding a text prompt to obtain a text encoding vector, and encoding a masked background image to obtain an image encoding vector, wherein the text prompt is used to describe the content of the generated image; concatenating the image encoding vector, a position mask, and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; progressively denoising the fused noise vector, and adding the text encoding vector and a texture guidance vector as guidance conditions during the denoising process to generate a filled image, wherein the texture guidance vector is obtained based on the concatenation, smoothing transition, and encoding processing of the texture features of the foreground image and the texture features of the background image.

[0132] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image filling method provided by the above methods. The method includes: encoding a text prompt word to obtain a text encoding vector, and encoding a masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the content of the generated image; concatenating the image encoding vector, a position mask, and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; progressively denoising the fused noise vector, and adding the text encoding vector and a texture guidance vector as guidance conditions during the denoising process to generate a filled image, wherein the texture guidance vector is obtained based on the concatenation, smoothing transition, and encoding processing of the texture features of the foreground image and the texture features of the background image.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image filling method, characterized in that, include: The text prompts are encoded to obtain a text encoding vector, and the masked background image is encoded to obtain an image encoding vector. The text prompts are used to describe the content of the generated image. The image encoding vector, the position mask, and random noise are concatenated to obtain a fused noise vector. The position mask is used to indicate the filling position of the foreground image in the background image. The fused noise vector is progressively denoised, and the text encoding vector and texture guiding vector are added as guiding conditions during the denoising process to generate a filled image; The steps for obtaining the texture guidance vector include: Texture features are extracted from the foreground image and the background image respectively to obtain foreground texture features and background texture features. The foreground texture features and the background texture features are then concatenated to obtain concatenated texture features. The splicing positions of the spliced ​​texture features are smoothed to obtain smooth texture features, and the smooth texture features are encoded to obtain the texture guiding vector; The number of targets and the degree of Gaussian blurring in the smooth transition process both decrease as the diffusion time step decreases.

2. The image filling method according to claim 1, characterized in that, The step of concatenating the foreground texture features and the background texture features to obtain concatenated texture features includes: Based on the location mask, the background texture features are masked to obtain the masked background texture features; Based on the mask size, the size of the foreground texture feature is adjusted to obtain the adjusted foreground texture feature; The adjusted foreground texture features and the masked background texture features are concatenated to obtain the concatenated texture features.

3. The image filling method according to claim 1, characterized in that, The process of smoothing the transition at the splicing positions of the spliced ​​texture features to obtain smooth texture features includes: The target number of pixels is expanded both inward and outward from the splicing position of the spliced ​​texture feature to obtain the ring-shaped region to be processed. The target number is determined based on the diffusion time step. Based on the diffusion time step, the annular region to be processed is subjected to Gaussian blurring to obtain the smooth transition texture feature.

4. The image filling method according to any one of claims 1 to 3, characterized in that, The step of concatenating the image encoding vector, the position mask, and random noise to obtain a fused noise vector includes: The location mask is downsampled to obtain the downsampled mask; The image encoding vector, the downsampled mask, and the random noise are concatenated to obtain a fused noise vector, wherein the image encoding vector, the downsampled mask, and the random noise have the same size.

5. The image filling method according to any one of claims 1 to 3, characterized in that, The stepwise denoising of the fused noise vector, incorporating the text encoding vector and texture guidance vector as guiding conditions during the denoising process, to generate a filled image includes: Based on the diffusion model, the text encoding vector and the texture guidance vector are applied to progressively denoise the fused noise vector to obtain a denoised feature vector. The text encoding vector is used as a text guidance condition input to the attention module of the encoder and decoder of the diffusion model, and the texture guidance vector is used as an image guidance condition input to the attention module of the encoder of the diffusion model. The denoised feature vectors are decoded to obtain the filled image.

6. An image filling device, characterized in that, include: The encoding unit is used to encode the text prompt words to obtain a text encoding vector, and to encode the masked background image to obtain an image encoding vector. The text prompt words are used to describe the generated image content. The splicing unit is used to splice the image encoding vector, the position mask, and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; The generation unit is used to perform stepwise denoising on the fused noise vector, and to add the text encoding vector and texture guidance vector as guidance conditions during the denoising process to generate a filled image; The steps for obtaining the texture guidance vector include: Texture features are extracted from the foreground image and the background image respectively to obtain foreground texture features and background texture features. The foreground texture features and the background texture features are then concatenated to obtain concatenated texture features. The splicing positions of the spliced ​​texture features are smoothed to obtain smooth texture features, and the smooth texture features are encoded to obtain the texture guiding vector; The number of targets and the degree of Gaussian blurring in the smooth transition process both decrease as the diffusion time step decreases.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image filling method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image filling method as described in any one of claims 1 to 5.

Images