Image filling method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides an image filling method and device, electronic equipment and a storage medium, and the method comprises the steps: encoding a text prompt word to obtain a text encoding vector, and encoding a masked background image to obtain an image encoding vector; splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector; and step-by-step denoising is carried out on the fused noise vector, a text coding vector and a texture guide vector are added in the denoising process as guide conditions, a filling image is generated, and the texture guide vector is obtained by carrying out splicing, smooth transition and coding processing on texture features of the foreground image and texture features of the background image. According to the method, the filling image which is rich in details, natural in texture and highly fused with the background image can be generated through a step-by-step denoising process and combining the text coding vector and the texture guide vector as double guide conditions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image filling method, device, electronic equipment and storage medium. Background Art

[0002] Image filling, as a key technology in the field of image processing, aims to restore or repair the occluded or missing parts of the image, and strive to make the repaired image visually natural and complete. This technology has a wide range of application scenarios and important commercial value, such as image restoration, background replacement, virtual fitting, and industrial defect synthesis.

[0003] Traditional image filling methods mainly rely on information such as color and texture around the target area, and complete the missing area through diffusion mechanism or sample-based methods. However, these methods are powerless when dealing with complex scenes. For example, when dealing with large-scale or complex-textured missing areas, diffusion-based methods often produce artifacts and blurring, resulting in unsatisfactory repair results. Sample-based methods are limited by computational efficiency and the ability to extract advanced features, making it difficult to achieve high-quality image repair while ensuring repair speed.

[0004] With the rapid development of artificial intelligence technology, especially the rise of artificial intelligence generated content (AICG) technology, deep learning-based image filling methods have emerged. These methods use advanced technologies such as convolutional neural networks (CNN) and generative adversarial networks (GAN) to capture the semantic features of image context and generate images. However, although these methods have made some progress in the field of image restoration, they still face many challenges. For example, how to balance the detailed features and overall structure of the image, how to achieve high-quality image filling in complex scenes, and how to ensure that the foreground of the filled image is highly consistent with the surrounding environment or that the foreground and background of the image transition naturally, are all issues that need to be solved urgently. Summary of the invention

[0005] The present invention provides an image filling method, device, electronic device and storage medium, which are used to solve the defects in the related art that the foreground consistency of the filled image is not high or the transition between the foreground and the background is unnatural, resulting in an unsatisfactory image filling effect.

[0006] The present invention provides an image filling method, comprising: Encoding the text prompt words to obtain a text encoding vector, and encoding the masked background image to obtain an image encoding vector, wherein the text prompt words are used to describe the generated image content; splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; The fused noise vector is gradually denoised, and the text encoding vector and texture guide vector are added as guiding conditions during the denoising process to generate a filling image, wherein the texture guide vector is obtained by splicing, smoothing and encoding the texture features of the foreground image and the texture features of the background image.

[0007] According to an image filling method provided by the present invention, the step of acquiring the texture guide vector comprises: Extracting texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and splicing the foreground texture features and the background texture features to obtain spliced ​​texture features; A smooth transition process is performed on the splicing position of the splicing texture feature to obtain a smooth texture feature, and the smooth texture feature is encoded to obtain the texture guide vector.

[0008] According to an image filling method provided by the present invention, the foreground texture feature and the background texture feature are spliced ​​to obtain the spliced ​​texture feature, including: Based on the position mask, masking is performed on the background texture feature to obtain a masked background texture feature; Based on the mask size, adjusting the size of the foreground texture feature to obtain an adjusted foreground texture feature; The adjusted foreground texture feature and the masked background texture feature are spliced ​​to obtain the spliced ​​texture feature.

[0009] According to an image filling method provided by the present invention, the step of performing smooth transition processing on the splicing position of the splicing texture feature to obtain the smooth texture feature includes: Expanding a target number of pixels inward and outward at the splicing position of the splicing texture feature to obtain a ring area to be processed, wherein the target number is determined based on a diffusion time step; Based on the diffusion time step, Gaussian blur processing is performed on the annular area to be processed to obtain the smooth transition texture feature.

[0010] According to an image filling method provided by the present invention, the number of targets decreases as the diffusion time step decreases, and the degree of Gaussian blurring of the annular area to be processed decreases as the diffusion time step decreases.

[0011] According to an image filling method provided by the present invention, the image coding vector, the position mask and the random noise are spliced ​​to obtain a fused noise vector, including: Downsampling the position mask to obtain a downsampled mask; The image coding vector, the downsampled mask and the random noise are concatenated to obtain a fused noise vector, wherein the image coding vector, the downsampled mask and the random noise have the same size.

[0012] According to an image filling method provided by the present invention, the fused noise vector is gradually denoised, and the text encoding vector and the texture guide vector are added as guide conditions during the denoising process to generate a filled image, including: Based on the diffusion model, the text encoding vector and the texture guiding vector are applied to gradually denoise the fused noise vector to obtain a denoised feature vector, the text encoding vector is input as a text guiding condition to the attention module of the encoder and decoder of the diffusion model, and the texture guiding vector is input as an image guiding condition to the attention module of the encoder of the diffusion model; The denoised feature vector is decoded to obtain a filled image.

[0013] The present invention also provides an image filling device, comprising: An encoding unit, used to encode the text prompt word to obtain a text encoding vector, and to encode the masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the generated image content; A splicing unit, used for splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used for indicating the filling position of the foreground image in the background image; A generating unit is used to gradually denoise the fused noise vector, and add the text encoding vector and texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained after splicing, smoothing transition and encoding of the texture features of the foreground image and the texture features of the background image.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements any of the above-mentioned image filling methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image filling method described above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image filling method described above is implemented.

[0017] The image filling method, device, electronic device and storage medium provided by the present invention can generate a filled image with rich details, natural texture and high fusion with the background image by gradually denoising the fused noise vector and combining the text encoding vector and the texture guide vector as dual guiding conditions in the denoising process. The introduction of the texture guide vector enables the texture features of the foreground image to smoothly transition with the background image, further enhancing the realism and visual beauty of the filled image. In addition, by encoding the text prompt words, the user's text description can be directly converted into specific guidance for image filling, increasing the flexibility and personalization of image generation, and by using a position mask to indicate the specific filling position of the foreground image in the background image, the image filling process is made more precise and controllable, thereby avoiding the problem of image misalignment or inaccurate filling area, and ensuring the visual consistency and coherence of the foreground and background of the generated filled image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic flow chart of the image filling method provided by the present invention; Figure 2 is a schematic diagram of the annular area to be processed provided by the present invention; Figure 3 is a flow chart of the texture-guided image filling method provided by the present invention; Figure 4 is a structural schematic diagram of an image filling device provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] Image inpainting is an image processing technology that aims to restore or repair the occluded or missing parts of the image to make it look as natural and complete as possible. Traditional image inpainting technology completes the target area through a diffusion mechanism or a sample-based method based on the color, texture and other information around the target area. In recent years, artificial intelligence technology, especially AIGC technology, has made rapid progress. New generative methods based on deep learning have emerged in an endless stream. Users' demand for image completion is not just to repair and complete the image features around the target area. A new application scenario has emerged: fill the missing area of ​​the background image or the specified target area with the user-specified foreground image content, so that the filled foreground image can retain consistency on the background image, and the transition between the foreground image and the background image can be highly harmonious.

[0022] At present, image filling methods are mainly divided into two categories: one is image filling based on traditional methods, and the other is generative methods based on deep learning. Traditional repair methods can be subdivided into methods based on diffusion mechanism, sample-based methods, and Poisson fusion methods. Image repair methods based on deep learning are mainly divided into image repair methods based on convolutional neural networks, image repair methods based on generative adversarial networks, and image filling methods based on diffusion models.

[0023] However, the above methods all have certain defects in practical application scenarios. The method based on diffusion mechanism mainly uses the edge information of the area to be repaired to determine the diffusion direction, and then smoothly diffuses to the edge and fills the known information. This type of method has a very impressive image repair effect when the damaged area is small, but when the missing area becomes larger or the texture is complex, the repair result will produce artifacts and blur. In addition, the repair results of this type of method also have defects such as inconsistency of structure and texture, and unreasonable repair content. The sample-based method calculates the similarity between the missing area and the known area, obtains the sample with the highest similarity, and copies it into the missing area to achieve the purpose of repairing the image. The calculation time of this type of method is too long and it is difficult to extract the high-level features of the image. Therefore, the image repair effect is not ideal, and it is difficult to ensure the rationality of the repaired image. The image fusion method based on Poisson fusion uses the Poisson equation in mathematics to achieve smooth transition and seamless fusion of foreground and background images. However, in some cases, Poisson fusion may affect the original color of the target image, resulting in visual incoherence between the fused area and the surrounding environment.

[0024] Image restoration methods based on convolutional neural networks use convolutional neural networks as feature extraction methods and capture semantic features through convolution operations. This type of method is mainly divided into two steps: encoding and decoding. The encoder is used to capture the contextual semantic features of the image, and the decoder uses these features to generate the missing image content. Image restoration methods based on generative adversarial networks are mainly composed of a generator and a discriminator. The goal of the generator is to generate an image, and the goal of the discriminator is to distinguish between the generated image and the real image. This type of method gradually generates realistic images through the continuous game between the generator and the discriminator, thereby achieving the purpose of repairing the image. The diffusion model is a generative model based on probability density function. This technology combines natural language processing and computer vision to achieve the transition from text description to image generation. It generates data by gradually diffusing random noise in the latent space. Compared with other generative models, the diffusion model has better generation quality and interpretability, so it has broad application prospects in the field of image generation.

[0025] However, current methods for image filling using diffusion models often only use one of the text guidance or image guidance, ignoring the complementarity of the two types of information. Using text guidance alone lacks control over the content generated by the diffusion model, while using image guidance alone ignores the large-scale semantic information used in model training, resulting in low consistency of the foreground after filling or unnatural transition between the foreground and background.

[0026] In this regard, the present invention proposes a texture-guided image filling method by combining the interpretability advantages of traditional methods with the high quality guarantee of the diffusion model effect. The Gabor operator is used to extract the texture features of the background image and the foreground image based on the pre-trained diffusion model. On the basis of retaining the high-order semantic information provided by the text guidance, the texture features of the spliced ​​foreground and background are used as additional diffusion model guidance conditions through the proposed texture transition module, so that the diffusion model can complete the image filling task more accurately and controllably, thereby overcoming the above defects.

[0027] It should be noted that the overall input of the method proposed in the present invention includes three parts: a text prompt word, a background image, and a position mask for generating a foreground and a foreground image, wherein the text prompt word is used to describe the content of the image finally generated. The method proposed in the present invention can be structurally divided into two modules, namely a pre-trained diffusion model and a texture guidance module, wherein the pre-trained diffusion model includes a text encoder, an image encoder, a denoising model U-Net and an image decoder, and the texture guidance module includes a Gabor filter, a texture transition module and an image encoder. The image filling method provided by the present invention will be introduced in detail below in combination with the system module.

[0028] Figure 1 is a flow chart of the image filling method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110, encoding the text prompt words to obtain a text encoding vector, and encoding the masked background image to obtain an image encoding vector, wherein the text prompt words are used to describe the generated image content.

[0029] Specifically, a text prompt is a descriptive text that indicates or describes the content of the final generated image. For example, in the image filling task, the text prompt may be "a cat on the floor". This text information provides semantic guidance for the image filling process, so that the generated filled image can meet or be close to the user's expectations.

[0030] Through the text encoder in the pre-trained diffusion model, the input text prompt word can be encoded to obtain a text encoding vector. Here, the text encoding vector refers to the vector representation obtained by encoding the text prompt word, which can capture the semantic information in the text and serve as a guiding condition in the image filling process (i.e., text guiding condition).

[0031] The masked background image is the image obtained by applying the position mask to the original background image. The position mask is a binary image (or grayscale image) of the same size as the background image, which is used to indicate the filling position of the foreground image in the background image. It defines which areas are foreground (areas that need to be filled) and which areas are background (areas that are retained). By applying this mask to the background image, some areas in the background image can be retained, while other areas (i.e., foreground areas) are set to 0 or a specific value for subsequent image filling.

[0032] Through the image encoder in the pre-trained diffusion model, the masked background image can be encoded to obtain an image encoding vector. Here, the image encoding vector refers to the vector representation obtained by encoding the masked background image. This vector representation can capture the visual features in the image, such as color, texture, shape, etc.

[0033] Step 120, concatenating the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image.

[0034] Specifically, after obtaining the image coding vector, the image coding vector, the position mask, and random noise of the same size can be concatenated to obtain a fused noise vector, and the vector is used as the input of the denoising model U-Net. Here, the fused noise vector is the result of the above concatenation operation, which contains all the information of the image coding vector, the position mask, and the random noise. This vector will be used as the input of the subsequent denoising process, and the filled image will be finally generated by stepwise denoising and adding the text coding vector and the texture guide vector as the guiding conditions.

[0035] It is understandable that this step is to generate a fused noise vector by splicing the image encoding vector, position mask and random noise. This vector contains image information, position information and random information. The introduction of random noise is to increase the diversity of generated images and avoid generating overly certain or repetitive images. By splicing this information together, the model can consider the image content, position information and randomness at the same time when generating images, thereby generating images that are more in line with expectations.

[0036] Step 130, gradually denoising the fused noise vector, and adding the text encoding vector and texture guide vector as guiding conditions in the denoising process to generate a filled image, wherein the texture guide vector is obtained by splicing, smoothing and encoding the texture features of the foreground image and the texture features of the background image.

[0037] Specifically, progressive denoising means that in the process of generating a filled image, through a series of steps or iterative operations, the noise or unnecessary details in the image are gradually reduced or eliminated, so as to obtain a clearer, smoother or desired image. In the denoising process, the text encoding vector and texture guidance vector are added to provide additional semantic and visual information to the denoising model U-Net, so as to guide the model to generate a more desired image.

[0038] Among them, the text encoding vector, as a text guidance condition, contains descriptive information about the content of the generated image. By incorporating this information into the denoising process, it can be ensured that the generated image is consistent with the content described by the text prompt word. The texture guidance vector, as an image guidance condition, is obtained by splicing, smoothing and encoding the texture features of the foreground image and the background image. This vector can capture the texture information in the image, thereby guiding the generation of image areas with similar textures. It should be understood that the texture guidance vector is a special vector representation that combines the texture features of the foreground image and the background image. By splicing, smoothing and encoding these features, a smooth vector containing both foreground texture information and background texture information can be obtained. This vector is used as a guidance condition in the denoising process to ensure that the generated image is consistent in texture with the foreground and background images.

[0039] Specifically, when the fused noise vector is gradually denoised, the fused noise vector can be used as input and passed to the denoising model U-Net. Through a series of iterative operations, the denoising model will gradually reduce or eliminate the noise in the image, which specifically involves operations such as extraction, transformation and reconstruction of image features. In each denoising process, the text encoding vector and texture guidance vector are added as guidance conditions. These vectors are combined with the current denoising result in a specific way to guide the generation of an image that is more in line with the expectations. For example, the text encoding vector can be used as a text guidance condition and input into the attention module of the encoder / decoder of the denoising model U-Net as the K value, while the texture guidance vector can be used as an image guidance condition and input into the attention module of the encoder of the denoising model U-Net as the V value. After multiple iterations, the denoising model outputs the final features after the T-step diffusion process (i.e., the denoising process), and the filled image can be obtained using the image decoder.

[0040] Here, the filled image refers to the image obtained by stepwise denoising and adding guiding conditions, which combines the contextual information of the background image and the texture features of the foreground image, as well as the semantic information described by the text prompt word. The generated filled image is visually consistent with the original image and its content conforms to the description of the text prompt word.

[0041] The method provided by the embodiment of the present invention can generate a filled image with rich details, natural texture and high integration with the background image by gradually denoising the fused noise vector and combining the text encoding vector and the texture guide vector as dual guiding conditions in the denoising process. The introduction of the texture guide vector enables the texture features of the foreground image to smoothly transition with the background image, further enhancing the realism and visual beauty of the filled image. In addition, by encoding the text prompt words, the user's text description can be directly converted into specific guidance for image filling, increasing the flexibility and personalization of image generation. By using a position mask to indicate the specific filling position of the foreground image in the background image, the image filling process is made more precise and controllable, thereby avoiding the problem of image misalignment or inaccurate filling area, and ensuring the visual consistency and coherence of the foreground and background of the generated filled image.

[0042] Based on the above embodiment, the step of acquiring the texture guide vector includes: Step 210 , extracting texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and splicing the foreground texture features and the background texture features to obtain spliced ​​texture features.

[0043] Specifically, the texture guidance module processes the foreground image and the background image accordingly to obtain the texture guidance vector. Specifically, the foreground image and the background image are respectively subjected to Gabor filters to extract multi-directional texture features as prior information. The Gabor filter here can be a 2D Gabor filter, which uses the Gabor kernel to filter the image to describe the local frequency information of the signal. The kernel function of the Gabor filter can be expressed as: in: and is the pixel position of the Gabor filter kernel, They represent the wavelength, angle, phase offset, standard deviation, and ellipticity of the filter kernel respectively.

[0044] Furthermore, in step 210, the foreground texture feature and the background texture feature are spliced ​​to obtain a spliced ​​texture feature, including: Step 211, performing mask processing on the background texture feature based on the position mask to obtain a masked background texture feature; Step 212, adjusting the size of the foreground texture feature based on the mask size to obtain an adjusted foreground texture feature; Step 213: splicing the adjusted foreground texture feature and the masked background texture feature to obtain the spliced ​​texture feature.

[0045] Specifically, after extracting the background texture and foreground texture features through the Gabor filter, before splicing the texture features of the two, corresponding preprocessing is required. Among them, the texture features extracted from the background image are multiplied by the position mask to obtain the masked background texture features. Then the texture features of the foreground are resized according to the mask size (referring to adjusting the size of the texture features to meet specific size requirements). This is because when splicing the texture features of the foreground and background, if their sizes are inconsistent, they cannot be spliced ​​directly. Therefore, the texture features of the foreground need to be adjusted to the same size as the mask (that is, the part of the background texture features that has been masked) for subsequent splicing operations. Subsequently, the resized foreground texture features are assigned to the masked background texture features to obtain the spliced ​​texture features.

[0046] Step 220 , performing smooth transition processing on the splicing position of the splicing texture feature to obtain a smooth texture feature, and encoding the smooth texture feature to obtain the texture guide vector.

[0047] It should be noted that directly splicing the texture features of the foreground and background is very likely to result in an unnatural transition at the splicing position. Therefore, the embodiment of the present invention designs a texture transition module related to the diffusion time step t based on the characteristics of the denoising model, which is used to perform Gaussian blur at the splicing position to make the texture features smooth at the splicing position and gradually reduce the texture connectivity restrictions at the splicing position as the diffusion time step t of the denoising model decreases.

[0048] Here, denoising models (also called diffusion models) are a type of generative models that generate new data by gradually adding noise to the data and then learning a reverse process (i.e., gradually generating data from the noise). In denoising models, a diffusion time step t refers to a point in time during the gradual transition from the original data to complete noise (or vice versa). In the forward diffusion process, t represents the number of steps or degree of adding noise; in the reverse diffusion (i.e., generation) process, t represents the number of steps or degree of gradually recovering the original data from the noise.

[0049] Furthermore, in step 220, the smooth transition processing is performed on the splicing position of the splicing texture feature to obtain a smooth texture feature, including: Step 221, expanding a target number of pixels inward and outward at the splicing position of the splicing texture feature to obtain a ring area to be processed, wherein the target number is determined based on a diffusion time step; Step 222: Based on the diffusion time step, Gaussian blur processing is performed on the annular area to be processed to obtain the smooth transition texture feature.

[0050] Figure 2 is a schematic diagram of the annular area to be processed provided by the present invention, such as Figure 2 As shown in the figure, the specific process of achieving smooth texture transition is as follows: for the time step t of the diffusion model, expand d pixels inside and outside the splicing position to obtain the annular area to be processed, where d is the target number, and its calculation formula is as follows: Where a and b are predefined parameters. As the time step t decreases, d gradually decreases, and the texture feature limit as an image guide also gradually decreases. In the 2d range of the image splicing position after the foreground and background are spliced, a Gaussian blur related to the time step t is performed. The Gaussian kernel is defined as: in and is the pixel position of the Gaussian kernel, is the standard deviation of the Gaussian kernel. As the time step t decreases, the content generated by the denoising model becomes clearer. Therefore, it is hoped that the degree of Gaussian blur at the splicing position will also decrease to enhance the texture guidance strength. Therefore, it is defined as: Where c is a hyperparameter, so the simplified Gaussian kernel is defined as: For each time step t, the concatenated texture features are processed by the texture transition module to obtain a smooth transition texture feature, which is input into the image encoder. The obtained vector is the texture guidance vector, which is used as the image guidance condition and input into the attention module of the encoder of the denoising model U-Net as the V value.

[0051] Based on any of the above embodiments, step 120 specifically includes: Step 121, downsampling the position mask to obtain a downsampled mask; Step 122, concatenating the image coding vector, the downsampled mask and the random noise to obtain a fused noise vector, wherein the image coding vector, the downsampled mask and the random noise have the same size.

[0052] Specifically, before splicing to obtain the fused noise vector, the position mask can be downsampled to make the size of the mask more closely match the size of the image coding vector, while reducing its resolution, removing redundant information, and reducing the computational complexity of subsequent processing steps. Subsequently, the image coding vector, the downsampled mask, and the random noise of the same size can be spliced. Here, the random noise is usually used to increase the diversity of the generated image or simulate uncertainty in the image, and it is adjusted to the same size as the coded vector and mask so that they can be spliced ​​together.

[0053] Specifically, before concatenation, it is necessary to ensure that the dimensions of the image encoding vector, the downsampled mask, and the random noise are consistent. If their dimensions are different, appropriate adjustments (such as padding, cropping, or adjusting the number of channels) can be made to match the dimensions. Subsequently, they can be concatenated along the channel dimension to form a comprehensive input containing image encoding information, position mask information, and random noise information. This comprehensive input can then be used as the input of the denoising model U-Net for subsequent processing.

[0054] Based on any of the above embodiments, step 130 specifically includes: Step 131, based on the diffusion model, applying the text encoding vector and the texture guidance vector, gradually denoising the fused noise vector to obtain a denoised feature vector, the text encoding vector is input as a text guidance condition to the attention module of the encoder and decoder of the diffusion model, and the texture guidance vector is input as an image guidance condition to the attention module of the encoder of the diffusion model; Step 132: Decode the denoised feature vector to obtain a filled image.

[0055] Specifically, after the fused noise vector is obtained by splicing, the vector is used as the input of the denoising model U-Net. At different time steps t, the encoder of the U-Net model will obtain the spliced ​​foreground and background texture features processed to different degrees, which are used as strong prior information to guide the output of each step of the diffusion model, and enhance the consistency of the diffusion model in retaining the specified foreground in the image filling task. The final features after the T-step diffusion process of the denoising model are decoded using the image decoder to obtain the filled image.

[0056] Based on any of the above embodiments, Figure 3 is a flow chart of the texture-guided image filling method provided by the present invention, such as Figure 3 As shown in the figure, the overall input of this method can be divided into three parts: (1) a text prompt word used to describe the content of the final generated image; (2) a background image and a position mask used to generate the foreground; (3) a foreground image.

[0057] The method proposed in the embodiment of the present invention can be structurally divided into two modules: (1) a pre-trained diffusion model, including a text encoder, an image encoder, a denoising model U-Net, and a final image decoder; (2) a texture-guided module, including a Gabor filter, a foreground-background texture transition module, and an image encoder.

[0058] For the texture guidance module, the foreground and background original images are separately subjected to Gabor filtering to extract multi-directional texture features as prior information. Then, the texture features extracted from the background are multiplied by the mask to obtain the masked background texture features. The foreground texture features are resized according to the mask size, and then the resized foreground texture features can be assigned to the masked background texture features to obtain the spliced ​​texture features.

[0059] Here, the direct splicing of the texture features of the foreground and background is very likely to result in an unnatural transition in the splicing position. Therefore, the embodiment of the present invention designs a texture transition module related to the diffusion time step t based on the characteristics of the denoising model U-Net, which is used to perform Gaussian blur at the splicing position to make the texture features smooth at the splicing position, and gradually reduce the texture connectivity restrictions at the splicing position as the diffusion process t of the diffusion model decreases. Specifically, for the time step t of the denoising model, d pixels are expanded inward and outward at the splicing position to obtain an annular area to be processed. As the time step t decreases, d gradually decreases, and the texture feature restrictions as image guidance also gradually decrease. Within the 2d range of the splicing position of the image after the foreground and background are spliced ​​(i.e., the annular area to be processed), a Gaussian blur related to the time step t is performed. For each time step t, the spliced ​​texture features are processed by the texture transition module and then input into the image encoder. The obtained vector is used as the image guidance condition and input into the attention module of the encoder of the denoising model U-Net as the V value. It should be noted that Figure 3 The intensity superposition strategy in refers to using the diffusion time step t to perform Gaussian blur processing on the splicing position, and as the time step t decreases, the degree of Gaussian blur also decreases, thereby enhancing the texture guidance strength.

[0060] For the pre-trained diffusion model part, the text encoder encodes the input text prompt word, and the encoded text vector is used as a text guidance condition to input the attention module of the encoder / decoder of the denoising model U-Net as the K value. The image encoder encodes the masked background image, and the encoded vector is spliced ​​with the downsampled mask and random noise of the same size as the input of the denoising model U-Net. At different time steps t, the U-Net encoder will obtain the spliced ​​foreground and background texture features after processing to different degrees, which are used as strong prior information to guide the output of each step of the denoising model, thereby enhancing the consistency of the denoising model in retaining the specified foreground in the image filling task. The final features after the T-step diffusion process of the denoising model are decoded using the image decoder to obtain the filled image. It should be understood that Figure 3 Z T represents the input of the denoising model U-Net at time step T, Z T-1 It represents the output result of the denoising model U-Net after the first denoising process, and Z0 is the output of the denoising model after time step T (that is, after T denoising processes).

[0061] The method provided in the embodiment of the present invention uses the Gabor operator to extract texture features of the background image and the foreground image based on a pre-trained diffusion model, and on the basis of retaining the high-order semantic information provided by the text guidance, uses the texture features of the splicing of the two as additional diffusion model guidance conditions through the proposed texture transition module, so that the diffusion model can complete the image filling task more accurately and controllably.

[0062] Based on any of the above embodiments, Figure 4 is a schematic diagram of the structure of the image filling device provided by the present invention, such as Figure 4 As shown, the device comprises: The encoding unit 410 is used to encode the text prompt word to obtain a text encoding vector, and encode the masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the generated image content; A splicing unit 420, configured to splice the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used to indicate a filling position of the foreground image in the background image; The generating unit 430 is used to gradually denoise the fused noise vector, and add the text encoding vector and the texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained after splicing, smoothing transition and encoding of the texture features of the foreground image and the texture features of the background image.

[0063] The device provided by the embodiment of the present invention can generate a filled image with rich details, natural texture and high integration with the background image by gradually denoising the fused noise vector and combining the text encoding vector and the texture guide vector as dual guiding conditions in the denoising process. The introduction of the texture guide vector enables the texture features of the foreground image to smoothly transition with the background image, further enhancing the realism and visual beauty of the filled image. In addition, by encoding the text prompt words, the user's text description can be directly converted into specific guidance for image filling, increasing the flexibility and personalization of image generation. By using a position mask to indicate the specific filling position of the foreground image in the background image, the image filling process is made more precise and controllable, thereby avoiding the problem of image misalignment or inaccurate filling area, and ensuring the visual consistency and coherence of the foreground and background of the generated filled image.

[0064] Based on any of the above embodiments, the device further includes a texture guiding unit, and the texture guiding unit includes: A feature extraction unit is used to extract texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and to splice the foreground texture features and the background texture features to obtain spliced ​​texture features; The texture transition unit is used to perform smooth transition processing on the splicing position of the splicing texture feature to obtain a smooth texture feature, and encode the smooth texture feature to obtain the texture guide vector.

[0065] Based on any of the above embodiments, the feature extraction unit includes a feature splicing subunit, and the feature splicing subunit is used to: Based on the position mask, masking is performed on the background texture feature to obtain a masked background texture feature; Based on the mask size, adjusting the size of the foreground texture feature to obtain an adjusted foreground texture feature; The adjusted foreground texture feature and the masked background texture feature are spliced ​​to obtain the spliced ​​texture feature.

[0066] Based on any of the above embodiments, the texture transition unit is specifically used for: Expanding a target number of pixels inward and outward at the splicing position of the splicing texture feature to obtain a ring area to be processed, wherein the target number is determined based on a diffusion time step; Based on the diffusion time step, Gaussian blur processing is performed on the annular area to be processed to obtain the smooth transition texture feature.

[0067] Based on any of the above embodiments, the number of targets decreases as the diffusion time step decreases, and the degree of Gaussian blur processing performed on the annular area to be processed decreases as the diffusion time step decreases.

[0068] Based on any of the above embodiments, the splicing unit 420 is specifically used for: Downsampling the position mask to obtain a downsampled mask; The image coding vector, the downsampled mask and the random noise are concatenated to obtain a fused noise vector, wherein the image coding vector, the downsampled mask and the random noise have the same size.

[0069] Based on any of the above embodiments, the generating unit 430 is specifically used for: Based on the diffusion model, the text encoding vector and the texture guiding vector are applied to gradually denoise the fused noise vector to obtain a denoised feature vector, the text encoding vector is input as a text guiding condition to the attention module of the encoder and decoder of the diffusion model, and the texture guiding vector is input as an image guiding condition to the attention module of the encoder of the diffusion model; The denoised feature vector is decoded to obtain a filled image.

[0070] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the image filling method, which includes: encoding the text prompt word to obtain a text encoding vector, and encoding the masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the generated image content; splicing the image encoding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; performing stepwise denoising on the fused noise vector, and adding the text encoding vector and the texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained after splicing, smoothing and encoding the texture features of the foreground image and the texture features of the background image.

[0071] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image filling method provided by the above methods, which includes: encoding text prompt words to obtain text encoding vectors, and encoding the masked background image to obtain image encoding vectors, wherein the text prompt words are used to describe the generated image content; splicing the image encoding vector, position mask and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; gradually denoising the fused noise vector, and adding the text encoding vector and texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained based on splicing, smoothing transition and encoding of the texture features of the foreground image and the texture features of the background image.

[0073] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the image filling method provided by the above-mentioned methods, the method comprising: encoding a text prompt word to obtain a text encoding vector, and encoding a masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the generated image content; splicing the image encoding vector, a position mask and random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; gradually denoising the fused noise vector, and adding the text encoding vector and the texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained after splicing, smoothing and encoding the texture features of the foreground image and the texture features of the background image.

[0074] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image filling method, characterized in that: include: Encoding the text prompt words to obtain a text encoding vector, and encoding the masked background image to obtain an image encoding vector, wherein the text prompt words are used to describe the generated image content; splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used to indicate the filling position of the foreground image in the background image; The fused noise vector is gradually denoised, and the text encoding vector and texture guide vector are added as guiding conditions during the denoising process to generate a filling image, wherein the texture guide vector is obtained by splicing, smoothing and encoding the texture features of the foreground image and the texture features of the background image.

2. The image filling method according to claim 1, characterized in that: The step of obtaining the texture guide vector comprises: Extracting texture features from the foreground image and the background image respectively to obtain foreground texture features and background texture features, and splicing the foreground texture features and the background texture features to obtain spliced ​​texture features; A smooth transition process is performed on the splicing position of the splicing texture feature to obtain a smooth texture feature, and the smooth texture feature is encoded to obtain the texture guide vector.

3. The image filling method according to claim 2, characterized in that: The step of splicing the foreground texture feature and the background texture feature to obtain a spliced ​​texture feature includes: Based on the position mask, masking is performed on the background texture feature to obtain a masked background texture feature; Based on the mask size, adjusting the size of the foreground texture feature to obtain an adjusted foreground texture feature; The adjusted foreground texture feature and the masked background texture feature are spliced ​​to obtain the spliced ​​texture feature.

4. The image filling method according to claim 2, characterized in that: The step of performing smooth transition processing on the splicing position of the splicing texture feature to obtain a smooth texture feature includes: Expanding a target number of pixels inward and outward at the splicing position of the splicing texture feature to obtain a ring area to be processed, wherein the target number is determined based on a diffusion time step; Based on the diffusion time step, Gaussian blur processing is performed on the annular area to be processed to obtain the smooth transition texture feature.

5. The image filling method according to claim 4, characterized in that: The number of targets decreases as the diffusion time step decreases, and the degree of Gaussian blurring of the annular region to be processed decreases as the diffusion time step decreases.

6. The image filling method according to any one of claims 1 to 5, characterized in that: The step of splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector includes: Downsampling the position mask to obtain a downsampled mask; The image coding vector, the downsampled mask and the random noise are concatenated to obtain a fused noise vector, wherein the image coding vector, the downsampled mask and the random noise have the same size.

7. The image filling method according to any one of claims 1 to 5, characterized in that: The step of gradually denoising the fused noise vector and adding the text encoding vector and the texture guide vector as guide conditions in the denoising process to generate a filled image includes: Based on the diffusion model, the text encoding vector and the texture guiding vector are applied to gradually denoise the fused noise vector to obtain a denoised feature vector, the text encoding vector is input as a text guiding condition to the attention module of the encoder and decoder of the diffusion model, and the texture guiding vector is input as an image guiding condition to the attention module of the encoder of the diffusion model; The denoised feature vector is decoded to obtain a filled image.

8. An image filling device, characterized in that: include: An encoding unit, used to encode the text prompt word to obtain a text encoding vector, and to encode the masked background image to obtain an image encoding vector, wherein the text prompt word is used to describe the generated image content; A splicing unit, used for splicing the image coding vector, the position mask and the random noise to obtain a fused noise vector, wherein the position mask is used for indicating the filling position of the foreground image in the background image; A generating unit is used to gradually denoise the fused noise vector, and add the text encoding vector and texture guide vector as guiding conditions in the denoising process to generate a filling image, wherein the texture guide vector is obtained after splicing, smoothing transition and encoding of the texture features of the foreground image and the texture features of the background image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image filling method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image filling method according to any one of claims 1 to 7 is implemented.

Images