Image exterior drawing method and device based on attention alignment and frequency decomposition

Abstract

The invention relates to the technical field of image processing, in particular to an image exterior drawing method and device based on attention alignment and frequency decomposition, and the method comprises the steps: determining a to-be-exterior-drawn region according to a binary mask of a source image; determining an initial latent variable of a reverse denoising process of the pre-training diffusion model, and adjusting attention distribution between the to-be-drawn region and the source image region by using an attention alignment index to obtain a first-stage optimized noise latent variable; and processing the first-stage optimized noise latent variable by adopting a preset noise re-initialization strategy based on frequency decomposition to fuse the low-frequency structure information and the high-frequency details to obtain a second-stage optimized noise latent variable, and obtaining an exterior drawing result based on the second-stage optimized noise latent variable. Therefore, the problem that due to the fact that the adaptability of the pre-training model to scenes and details is limited, semantic inconsistency, spatial dislocation and high-frequency artifacts are prone to occurring in the generation area, and the authenticity and visual quality of the exterior drawing effect are affected is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image out-drawing method and device based on attention alignment and frequency decomposition. BACKGROUND

[0002] Image out-drawing refers to using an algorithm to infer and generate visual content beyond the original image boundary under the condition of only given part of the image content, so as to realize the expansion, completion or scene extension of the image.

[0003] In related technologies, a pre-trained model is usually used for image out-drawing, which can understand the image content at the semantic level and thus generate a more natural boundary extension effect to effectively complete the source image region.

[0004] However, due to the limitation of data distribution and scene diversity in the training process of the pre-trained model, the generated result is prone to be inconsistent with the source image in semantics or spatial position, and in addition, high-frequency artifacts and abnormal textures are prone to occur in the generated region, which reduces the naturalness and authenticity of the overall image and affects the application value of the out-drawing effect, which needs to be solved urgently. SUMMARY

[0005] The present application provides an image out-drawing method and device based on attention alignment and frequency decomposition to solve the problem that in related technologies, due to the limited adaptability of the pre-trained model to scenes and details, the generated region is prone to have semantic inconsistency, spatial misplacement and high-frequency artifacts, thereby affecting the authenticity and visual quality of the out-drawing effect.

[0006] The first aspect embodiment of the present application provides an image out-drawing method based on attention alignment and frequency decomposition, comprising the following steps: determining a to-be-out-drawn region according to a binary mask of a source image; determining an initial latent variable of a reverse denoising process of a pre-trained diffusion model, and using an attention alignment index to iteratively optimize the initial latent variable to adjust the attention distribution between the to-be-out-drawn region and the source image region until the iteration ends, to obtain a first-stage optimized noise latent variable; using a preset noise reinitialization strategy based on frequency decomposition to process the first-stage optimized noise latent variable to fuse low-frequency structure information satisfying a preset robust condition and high-frequency details satisfying a preset reasonable condition, to obtain a second-stage optimized noise latent variable, and obtaining an out-drawing result based on the second-stage optimized noise latent variable.

[0007] By means of the above technical means, the initial latent variable can be iteratively optimized by the attention alignment index in the embodiments of the present application, so as to calibrate the attention distribution between the to-be-outlined region and the source image region, and then based on the preset noise reinitialization strategy of frequency decomposition, the low-frequency structure information and the high-frequency details are fused to obtain the second-stage optimized noise latent variable, which is output to the pre-trained diffusion model to generate the outliner result, so that the outliner region is consistent in semantics with the source image, the global structure uniformity and the local texture natural details are taken into account, the high-frequency artifacts are effectively suppressed, the visual authenticity and the overall aesthetic sense are improved, and the high-quality and natural outliner effect is realized.

[0008] Optionally, in an embodiment of the present application, the outliner result based on the second-stage optimized noise latent variable comprises: inputting the second-stage optimized noise latent variable into the pre-trained diffusion model to perform a complete reverse denoising process to output the outliner result containing complete image content.

[0009] By means of the above technical means, the outliner result containing complete image content can be output by using the second-stage optimized noise latent variable instead of directly using random noise, so that the semantics and structure of the outliner region can be more accurately controlled, the generated content is more natural and detailed, and the overall visual authenticity and consistency are significantly improved.

[0010] Optionally, in an embodiment of the present application, the to-be-outlined region is determined according to a binary mask of the source image, comprising: obtaining the source image; constructing a binary mask corresponding to the source image; and encoding the source image and the binary mask into a model input format to determine the to-be-outlined region.

[0011] By means of the above technical means, the source image region and the to-be-outlined region can be explicitly distinguished according to the binary mask, so that the to-be-outlined region is determined, and different latent variable initialization and optimization strategies are adopted for different regions in the generation process, so as to ensure the high consistency of the outliner content with the source image in semantics, structure and visual effect.

[0012] Optionally, in an embodiment of the present application, the initial latent variable of the reverse denoising process of the pre-trained diffusion model comprises: setting the time step number of the diffusion process; sampling a random noise tensor from a preset standard Gaussian distribution according to the time step number; and determining the initial latent variable according to the noise tensor.

[0013] By means of the above technical means, the random noise tensor is sampled from the standard Gaussian distribution as the initial latent variable in the embodiments of the present application, which can provide diversified initial input for the image outliner process, while ensuring the uniformity and randomness of the latent variable in the statistical sense, laying a foundation for subsequent attention alignment optimization and frequency decomposition processing.

[0014] Optionally, in one embodiment of this application, the step of iteratively optimizing the initial latent variable using an attention alignment metric to adjust the attention distribution between the region to be mapped and the source image region until the iteration ends, thereby obtaining the first-stage optimized noise latent variable, includes: inputting the initial latent variable into the denoising network of the pre-trained diffusion model to extract the self-attention map of the denoising network, and calculating the attention of the region to be mapped based on the self-attention map; calculating the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region based on the attention of the region to be mapped and the binary mask, respectively; calculating the attention alignment metric based on the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region; using a preset gradient descent algorithm to iteratively update the mean and standard deviation of the Gaussian distribution followed by the initial input latent variable with the goal of minimizing the attention alignment metric, until the attention alignment metric meets a preset convergence condition, thereby obtaining the first-stage optimized noise latent variable.

[0015] Through the above technical means, the embodiments of this application can adjust the attention distribution between the region to be drawn and the source image region according to the attention alignment index to obtain the first-stage optimized noise latent variable. This can make the boundary and contour of the drawn region naturally connect with the source image, avoid the abrupt or discontinuous phenomenon in the model generation result, and better match the local features of the existing region of the source image, reduce high-frequency artifacts and unnatural texture generation, thereby improving the overall coherence and visual realism of the drawn result.

[0016] Optionally, in one embodiment of this application, the step of processing the first-stage optimized noise latent variables using a preset noise re-initialization strategy based on frequency decomposition to fuse low-frequency structural information satisfying preset robustness conditions with high-frequency details satisfying preset reasonableness conditions to obtain second-stage optimized noise latent variables includes: inputting the first-stage optimized noise latent variables into the pre-trained diffusion model, performing a reverse denoising process to generate a preliminary external drawing image result, and applying a forward noise addition process to the preliminary external drawing image result to generate an initial noise tensor containing low-frequency structural information; and applying a fast Fourier transform to the initial noise tensor. The low-frequency structural information satisfying the preset robustness condition is extracted using a preset low-pass filter; a fast Fourier transform is applied to random variables sampled from a preset standard Gaussian distribution, and a preset high-pass filter is used to extract the high-frequency details satisfying the preset reasonable condition; the low-frequency structural information satisfying the preset robustness condition and the high-frequency details satisfying the preset reasonable condition are weighted and superimposed to obtain a fused mixed noise tensor; based on the binary mask, the mixed noise tensor is applied to the region to be drawn, and the first-stage optimized noise latent variable of the source image region is retained to obtain the second-stage optimized noise latent variable.

[0017] Through the above technical means, the embodiments of this application can process low-frequency components and high-frequency components separately based on the preset noise re-initialization strategy of frequency decomposition. The low-frequency components can be used to maintain the overall structural robustness of the source image, while the high-frequency components can be used to introduce reasonable details and textures, thereby ensuring the consistency of the global structure when generating the external drawing result and improving the naturalness and visual realism of local details.

[0018] A second aspect of this application provides an image outlining device based on attention alignment and frequency decomposition, comprising: a determination module for determining an outlining region based on a binary mask of a source image; an optimization module for determining initial latent variables in the reverse denoising process of a pre-trained diffusion model, and iteratively optimizing the initial latent variables using an attention alignment index to adjust the attention distribution between the outlining region and the source image region until the iteration ends, thereby obtaining a first-stage optimized noise latent variable; and an outlining module for processing the first-stage optimized noise latent variable using a preset noise re-initialization strategy based on frequency decomposition to fuse low-frequency structural information that satisfies preset robustness conditions with high-frequency details that satisfy preset reasonable conditions, thereby obtaining a second-stage optimized noise latent variable, and obtaining an outlining result based on the second-stage optimized noise latent variable.

[0019] Through the above technical means, the embodiments of this application can iteratively optimize the initial latent variables through the attention alignment index to calibrate the attention distribution between the region to be drawn and the source image region. Then, based on the preset noise re-initialization strategy of frequency decomposition, low-frequency structural information and high-frequency details are fused to obtain the second-stage optimized noise latent variables, which are output to the pre-trained diffusion model to generate the drawing result. This can make the drawing region semantically consistent with the source image, take into account the uniformity of the global structure and the natural details of the local texture, effectively suppress high-frequency artifacts, improve visual realism and overall aesthetics, and achieve a high-quality, naturally extended drawing effect.

[0020] Optionally, in one embodiment of this application, the external drawing module includes: an output unit, used to input the second-stage optimized noise latent variable into the pre-trained diffusion model, perform a complete reverse denoising process, and output the external drawing result containing complete image content.

[0021] Through the above technical means, the embodiments of this application can utilize the second-stage optimization of noise latent variables to output the external drawing result containing complete image content, instead of directly using random noise. This allows for more precise control of the semantics and structure of the external drawing region, making the generated content more natural, richer in detail, and significantly improving the overall visual realism and consistency.

[0022] Optionally, in one embodiment of this application, the determining module includes: an acquisition unit for acquiring the source image; a construction unit for constructing a binary mask corresponding to the source image; and a first determining unit for jointly encoding the source image and the binary mask into a model input format to determine the region to be drawn.

[0023] Through the above technical means, the embodiments of this application can explicitly distinguish between the source image region and the region to be drawn based on the binary mask, thereby clarifying the region to be drawn. In the generation process, differentiated latent variable initialization and optimization strategies are adopted for different regions to ensure that the drawn content is highly consistent with the source image in terms of semantics, structure and visual effects.

[0024] Optionally, in one embodiment of this application, the optimization module includes: a setting unit for setting the time step number of the diffusion process; a sampling unit for sampling a random noise tensor from a preset standard Gaussian distribution according to the time step number; and a second determining unit for determining the initial latent variable based on the noise tensor.

[0025] Through the above technical means, the embodiments of this application sample random noise tensors from the standard Gaussian distribution as initial latent variables, which can provide diverse initial inputs for the image drawing process, while ensuring the uniformity and randomness of the latent variables in a statistical sense, laying the foundation for subsequent attention alignment optimization and frequency decomposition processing.

[0026] Optionally, in one embodiment of this application, the optimization module includes: a first calculation unit, configured to input the initial latent variable into the denoising network of the pre-trained diffusion model to extract the self-attention map of the denoising network, and to calculate the attention of the region to be mapped based on the self-attention map; a second calculation unit, configured to calculate the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region based on the attention of the region to be mapped and the binary mask, respectively; a third calculation unit, configured to calculate the attention alignment index based on the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region; and an optimization unit, configured to use a preset gradient descent algorithm to iteratively update the mean and standard deviation of the Gaussian distribution followed by the initial input latent variable with the goal of minimizing the attention alignment index, until the attention alignment index meets a preset convergence condition, thereby obtaining the first-stage optimized noise latent variable.

[0027] Through the above technical means, the embodiments of this application can adjust the attention distribution between the region to be drawn and the source image region according to the attention alignment index to obtain the first-stage optimized noise latent variable. This can make the boundary and contour of the drawn region naturally connect with the source image, avoid the abrupt or discontinuous phenomenon in the model generation result, and better match the local features of the existing region of the source image, reduce high-frequency artifacts and unnatural texture generation, thereby improving the overall coherence and visual realism of the drawn result.

[0028] Optionally, in one embodiment of this application, the external drawing module includes: a generation unit, configured to input the first-stage optimized noise latent variable into the pre-trained diffusion model, perform a reverse denoising process to generate a preliminary external drawing image result, and apply a forward noise addition process to the preliminary external drawing image result to generate an initial noise tensor containing low-frequency structural information; a first extraction unit, configured to apply a fast Fourier transform to the initial noise tensor and extract the low-frequency structural information that meets the preset robustness conditions through a preset low-pass filter; a second extraction unit, configured to sample random variables from a preset standard Gaussian distribution, apply a fast Fourier transform, and extract the high-frequency details that meet the preset reasonable conditions through a preset high-pass filter; a superposition unit, configured to perform weighted superposition and fusion of the low-frequency structural information that meets the preset robustness conditions and the high-frequency details that meet the preset reasonable conditions to obtain a fused mixed noise tensor; and an acquisition unit, configured to apply the mixed noise tensor to the region to be drawn based on the binary mask, and retain the first-stage optimized noise latent variable of the source image region to obtain the second-stage optimized noise latent variable.

[0029] Through the above technical means, the embodiments of this application can process low-frequency components and high-frequency components separately based on the preset noise re-initialization strategy of frequency decomposition. The low-frequency components can be used to maintain the overall structural robustness of the source image, while the high-frequency components can be used to introduce reasonable details and textures, thereby ensuring the consistency of the global structure when generating the external drawing result and improving the naturalness and visual realism of local details.

[0030] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image outlining method based on attention alignment and frequency decomposition as described in the above embodiments.

[0031] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image outlining method based on attention alignment and frequency decomposition.

[0032] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described image extrapolation method based on attention alignment and frequency decomposition.

[0033] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0034] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an image outlining method based on attention alignment and frequency decomposition provided according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the principle of an image outlining method based on attention alignment and frequency decomposition according to an embodiment of this application; Figure 3 This is a block diagram of an image outlining device based on attention alignment and frequency decomposition according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0035] Figure label: 10-Image drawing device based on attention alignment and frequency decomposition; 100-Determination module, 200-Optimization module, 300-Drawing module; 401-Memory, 402-Processor, 403-Communication interface. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] The image outlining method and apparatus based on attention alignment and frequency decomposition according to embodiments of this application are described below with reference to the accompanying drawings. To address the technical problem mentioned in the background that the limited adaptability of pre-trained models to scenes and details easily leads to semantic inconsistencies, spatial misalignments, and high-frequency artifacts in the generated regions, thus affecting the realism and visual quality of the outlining effect, this application provides an image outlining method based on attention alignment and frequency decomposition. In this method, the attention distribution between the region to be outlined and the source image region is adjusted through an attention alignment index to obtain the first-stage optimized noise latent variable. Then, based on a preset noise re-initialization strategy of frequency decomposition, low-frequency structural information and high-frequency details are fused to obtain the second-stage optimized noise latent variable, which is input into the pre-trained diffusion model to generate the outlining result. This allows for fine-tuning and reconstruction of the initial noise latent variable, guiding the generation process, thereby enabling the pre-trained model to make full use of contextual information, improve semantic consistency, effectively suppress high-frequency artifacts, and enhance the visual realism of the outlining image, significantly improving realism, coherence, and visual aesthetics. At the same time, no training or fine-tuning of the pre-trained model is required, and it can be seamlessly integrated with various mainstream pre-trained models as a plug-and-play module, thus greatly reducing the application threshold. This solves the problem that the limited adaptability of pre-trained models to scenes and details can easily lead to semantic inconsistencies, spatial misalignments, and high-frequency artifacts in the generated regions, thus affecting the realism and visual quality of the rendering effect.

[0038] Specifically, Figure 1 This is a flowchart illustrating an image outlining method based on attention alignment and frequency decomposition provided in an embodiment of this application.

[0039] like Figure 1 As shown, this image outlining method based on attention alignment and frequency decomposition includes the following steps: In step S101, the area to be drawn is determined based on the binary mask of the source image.

[0040] The source image can be the original input image; the binary mask can be a binary matrix of the same size as the source image, which can be used to identify the image. Its pixel values ​​are usually represented by 0 and 1, where 1 can correspond to the area to be drawn and 0 can correspond to the retained source image area; the area to be drawn can be the missing part of the image or the part that needs to be expanded and generated by the binary mask.

[0041] Optionally, in one embodiment of this application, determining the region to be drawn based on the binary mask of the source image includes: acquiring the source image; constructing a binary mask corresponding to the source image; and jointly encoding the source image and the binary mask into a model input format to determine the region to be drawn.

[0042] As a specific example, the embodiments of this application can first obtain a size of Source image And construct the corresponding binary mask. ;in It can represent pixel position ( i, j The area marked with ) is the area to be drawn. This can be represented as the preserved source image region; Furthermore, in this embodiment, the source image and the binary mask can be jointly encoded into the input format of the pre-trained diffusion model through channel splicing, weighted fusion, etc., and the pixel values ​​of the source image can be standardized to the [-1,1] interval through linear normalization, centering normalization, etc., to meet the input requirements of the pre-trained diffusion model; wherein, the binary mask can be used to explicitly distinguish the source image region and the region to be drawn during the optimization process.

[0043] In step S102, the initial latent variables of the reverse denoising process of the pre-trained diffusion model are determined, and the initial latent variables are iteratively optimized using the attention alignment index to adjust the attention distribution between the region to be drawn and the source image region until the iteration ends, thus obtaining the first-stage optimized noise latent variables.

[0044] It can be explained that the pre-trained diffusion model is a type of deep learning model based on the principle of probabilistic generation. Through pre-training on large-scale image datasets, it learns the step-by-step inversion process from noise to a clear image. During the training phase, the model learns the corresponding denoising steps by progressively adding noise to real images, thereby mastering the latent features of image distribution. During the inference phase, the model takes random noise or latent variables as input and generates image content that conforms to semantics and structure by progressively denoising according to the inverse diffusion process. This model can serve as a basic generative network in image outlining tasks, utilizing its powerful prior knowledge and semantic understanding capabilities to perform content completion and structural reconstruction of the region to be outlined, thereby generating an image that is consistent with the original. Figure 1 The result is a smooth and natural-looking external rendering. The inverse denoising process is a core step in the diffusion process of the pre-trained diffusion model. It achieves the inversion and reconstruction from the noise distribution to the true image distribution by progressively denoising the initial random noise or latent variables. The initial latent variables can be random representations used to initiate the inverse denoising process in the diffusion model, typically consisting of random noise tensors sampled from a standard Gaussian distribution.

[0045] In the embodiments of this application, the attention alignment metric can be a measure of the consistency between the attention distribution of the pre-trained diffusion model and the semantics of the source image during the generation process. It can evaluate the alignment degree of the model in feature reconstruction and semantic extension by analyzing the correlation between the internal attention mechanism of the model in the source image region and the region to be drawn.

[0046] Optionally, in one embodiment of this application, determining the initial latent variables for the reverse denoising process of the pre-trained diffusion model includes: setting the time step number of the diffusion process; sampling random noise tensors from a preset standard Gaussian distribution based on the time step number; and determining the initial latent variables based on the noise tensors.

[0047] The pre-defined standard Gaussian distribution can be a symmetric normal distribution with a mean of 0 and a variance of 1. It serves as the fundamental distribution for generating initial latent variables and noise tensors in the diffusion model, ensuring the randomness and controllability of model sampling. The noise tensor can be a multidimensional array used in the diffusion model or image generation task to represent random perturbations. Each element is typically sampled independently from the standard Gaussian distribution and can serve as an initial latent variable in the diffusion process or a random input to the drawn region. During the reverse denoising process, it is gradually mapped into an image with semantic and structural information.

[0048] Specifically, in this embodiment of the application, the time step of the diffusion process can be set to T, and further, from the standard Gaussian distribution... Medium-sampled random noise tensor , which serves as the initial latent variable for the diffusion model at the final time step T; where C, H, and W represent the number of channels, height, and width, respectively, and C is usually equal to 3.

[0049] Next, embodiments of this application can use the noise tensor With mask M, source image The pre-trained diffusion model is fed into the image mapping task as the initial input. It can be trained without additional supervision and the noise tensor can be used for subsequent optimization and reverse generation.

[0050] Optionally, in one embodiment of this application, the initial latent variables are iteratively optimized using an attention alignment metric to adjust the attention distribution between the region to be mapped and the source image region until the iteration ends, resulting in the first-stage optimized noise latent variables. This includes: inputting the initial latent variables into the denoising network of a pre-trained diffusion model to extract the self-attention map of the denoising network, and calculating the attention of the region to be mapped based on the self-attention map; calculating the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region based on the attention of the region to be mapped and the binary mask, respectively; calculating the attention alignment metric based on the average attention score of the region to be mapped to itself and the average attention score of the region to be mapped to the source image region; using the minimization of the attention alignment metric as the optimization objective, employing a preset gradient descent algorithm to iteratively update the mean and standard deviation of the Gaussian distribution followed by the initial input latent variables until the attention alignment metric meets the preset convergence condition, thus obtaining the first-stage optimized noise latent variables.

[0051] Understandably, a pre-defined gradient descent algorithm can update parameters in the opposite direction of the objective function's gradient, thereby gradually minimizing the loss function and balancing convergence speed, stability, and computational cost. Depending on the update strategy and gradient calculation method, the pre-defined gradient descent algorithm can include, but is not limited to, standard gradient descent, stochastic gradient descent, mini-batch gradient descent, etc., which can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here. The pre-defined convergence condition can be that the change in the index or the gradient is below a set threshold, indicating that the alignment has stabilized; or it can be reaching the maximum number of iterations to ensure that the optimization process terminates to a limited extent.

[0052] The following is a specific example, such as Figure 2 As shown in (b), the attention alignment process of this application embodiment will be described, which may include the following steps: (1) The embodiments of this application can be used to initialize the latent variables. The input is fed into the denoising network of the pre-trained diffusion model, and according to the initial time step set in the denoising process, the self-attention map calculated by the self-attention module inside the network is extracted, and the attention in the out-of-drawing region is extracted. It can be represented as: , in, x , y The horizontal and vertical coordinates represent the spatial location, respectively. A x,y This represents the fraction of self-attention corresponding to the spatial location (x,y). M A [ x, y [] represents a binary mask whose value can be used to indicate whether the spatial location (x, y) belongs to the region that needs to be drawn. WhenM A [ x, y When ]=1, it indicates that the position belongs to the drawn area; when M A [ x, y When ]=0, it indicates that the location belongs to the source image region.

[0053] (2) Further, in the embodiments of this application, the average attention score of the region to be drawn to itself can be calculated based on the self-attention map and the binary mask M. It can be represented as: , And calculate the average attention score of the region to be drawn to the source image region. It can be represented as: .

[0054] (3) Based on the above two attention scores, the embodiments of this application can calculate AAM (Attention AlignmentMetric) to measure the rationality of the attention distribution in the region to be drawn, which can be expressed as: , Where ϵ is a small positive constant to prevent the denominator from being zero, λ is the weight hyperparameter of the regularization term, and τ is the weight of the regularization term. The lower bound constraint applied.

[0055] (4) As one possible implementation method, the embodiments of this application can take minimizing the AAM index as the optimization objective, and use a gradient descent algorithm to iteratively update the mean and standard deviation of the Gaussian distribution followed by the initial latent variables until the index meets the set convergence condition, thereby obtaining the first-stage optimized noise latent variables. .

[0056] In step S103, a preset noise re-initialization strategy based on frequency decomposition is used to process the first-stage optimized noise latent variables, so as to fuse low-frequency structural information that meets preset robust conditions and high-frequency details that meet preset reasonable conditions to obtain the second-stage optimized noise latent variables, and to obtain the external drawing results based on the second-stage optimized noise latent variables.

[0057] Among them, the pre-set noise re-initialization based on frequency decomposition can separate and process low-frequency and high-frequency components in the frequency domain to generate robust low-frequency structures and reasonable high-frequency details, thereby providing better initial latent variables for the area to be drawn and improving the stability and visual effect of the diffusion model generation.

[0058] It can be noted that the preset robust conditions may include, but are not limited to, consistency with the low frequency of the source image, structural stability, and global semantic consistency; the preset reasonable conditions may include, but are not limited to, local semantic matching, natural texture, and smooth connection with the low frequency structure. These can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0059] Optionally, in one embodiment of this application, a preset noise re-initialization strategy based on frequency decomposition is used to process the first-stage optimized noise latent variables to fuse low-frequency structural information that meets preset robustness conditions and high-frequency details that meet preset reasonableness conditions to obtain the second-stage optimized noise latent variables. This includes: inputting the first-stage optimized noise latent variables into a pre-trained diffusion model, performing a reverse denoising process to generate a preliminary external drawing image result, and applying a forward noise addition process to the preliminary external drawing image result to generate an initial noise tensor containing low-frequency structural information; applying a fast Fourier transform to the initial noise tensor and passing it through a preset low-pass filter to extract low-frequency structural information that meets preset robustness conditions; sampling random variables from a preset standard Gaussian distribution, applying a fast Fourier transform, and extracting high-frequency details that meet preset reasonableness conditions through a preset high-pass filter; weighted superposition and fusion of the low-frequency structural information that meets preset robustness conditions and the high-frequency details that meet preset reasonableness conditions to obtain a fused mixed noise tensor; and applying the mixed noise tensor to the region to be drawn based on a binary mask, while retaining the first-stage optimized noise latent variables of the source image region to obtain the second-stage optimized noise latent variables.

[0060] The preset low-pass filter can be configured with different filter kernels based on the image resolution, the size of the drawn area, and the desired smoothness. For example, 3×3 or 5×5 convolution kernels can be used for spatial domain smoothing. Alternatively, a cutoff frequency (usually 10% to 30% of the maximum frequency of the image) can be set in the frequency domain to preserve low-frequency coefficients and obtain robust low-frequency structural information. The preset high-pass filter can also be configured based on the image resolution, the size of the drawn area, and the desired smoothness, without specific limitations.

[0061] The following is a specific example, such as Figure 2 As shown in (c), the frequency decomposition process obtained in the embodiments of this application will be described, which may include the following steps: (1) The embodiments of this application can optimize the noise latent variables in the first stage. The input is fed into a pre-trained diffusion model, where a complete reverse denoising process is performed to generate a preliminary external image. Subsequently, a positive noise addition process is applied to the preliminary result to generate an initial noise tensor containing low-frequency structural information. .

[0062] (2) In the embodiments of this application, an FFT (Fast Fourier Transform) can be applied to the initial noise tensor to transform it to the frequency domain, which can be expressed as: , in, This represents the initial noise tensor after the FFT; This represents the initial noise tensor containing low-frequency structural information.

[0063] Furthermore, embodiments of this application can extract low-frequency structural information using a low-pass filter to preserve structural semantics, which can be represented as: , in, Indicates low-frequency structural information; This represents a two-dimensional Gaussian low-pass filter used to filter out low-frequency information.

[0064] (3) In this embodiment of the application, a random variable can be sampled from a standard Gaussian distribution and subjected to a fast Fourier transform, which can be expressed as: , in, This represents noise from random sampling.

[0065] Furthermore, embodiments of this application can extract high-frequency details using a high-pass filter. It is used to provide texture detail and randomness; among which, This represents a two-dimensional Gaussian high-pass filter used to filter out high-frequency information.

[0066] (4) In the frequency domain, the embodiments of this application can perform weighted superposition and fusion of the low-frequency structural information and high-frequency details obtained above, and then restore it to the image space through IFFT (Inverse Fast Fourier Transform) to obtain a fused mixed noise tensor, which can be expressed as: , in, This represents the second stage of optimizing the noise latent variable.

[0067] (5) Based on the binary mask constructed in the above steps, the embodiments of this application can apply the mixed noise tensor to the region to be drawn, while retaining the first-stage optimized noise latent variable for the source image region, thereby constituting the second-stage optimized noise latent variable, which can be expressed as: , Optionally, in one embodiment of this application, obtaining the external drawing result based on the second-stage optimized noise latent variables includes: inputting the second-stage optimized noise latent variables into a pre-trained diffusion model, performing a complete reverse denoising process, and outputting an external drawing result containing complete image content.

[0068] Specifically, such as Figure 2 As shown in (a), embodiments of this application can optimize the noise latent variable in the second stage. As the initial input to the diffusion process, the combined preliminary external mapping results The mask M is used as a conditional input and is fed into the pre-trained diffusion model. Furthermore, in the embodiments of this application, a complete back diffusion process is performed, iteratively denoising from time step T to time step t=0, gradually predicting intermediate latent variables, and finally generating a complete output image.

[0069] Finally, in the embodiments of this application, a complete external drawing image result can be formed based on the binary mask constructed above, which includes the filled external drawing area and the source image area that remains unchanged.

[0070] The image outlining method based on attention alignment and frequency decomposition proposed in this application adjusts the attention distribution between the region to be outlined and the source image region through attention alignment index to obtain the first-stage optimized noise latent variable. Then, based on the preset noise re-initialization strategy of frequency decomposition, low-frequency structural information and high-frequency details are fused to obtain the second-stage optimized noise latent variable, which is input into the pre-trained diffusion model for image outlining. This method can finely optimize and reconstruct the initial input noise, guide the generation process, enable the pre-trained model to make full use of contextual information, improve semantic consistency, effectively suppress high-frequency artifacts, and enhance the visual realism of the outlined image, significantly improving realism, coherence, and visual aesthetics. At the same time, no training or fine-tuning of the pre-trained model is required, and it can be seamlessly integrated with various mainstream pre-trained models as a plug-and-play module, greatly reducing the application threshold.

[0071] Next, referring to the accompanying drawings, an image outlining apparatus based on attention alignment and frequency decomposition proposed according to an embodiment of this application is described.

[0072] Figure 3 This is a block diagram of an image rendering device based on attention alignment and frequency decomposition according to an embodiment of this application.

[0073] like Figure 3 As shown, the image rendering device 10 based on attention alignment and frequency decomposition includes: a determination module 100, an optimization module 200, and a rendering module 300.

[0074] The determining module 100 is used to determine the region to be drawn based on the binary mask of the source image.

[0075] The optimization module 200 is used to determine the initial latent variables in the reverse denoising process of the pre-trained diffusion model, and to iteratively optimize the initial latent variables using the attention alignment index to adjust the attention distribution between the region to be drawn and the source image region until the iteration ends, thus obtaining the first-stage optimized noise latent variables.

[0076] The external drawing module 300 is used to process the first-stage optimized noise latent variables using a preset noise re-initialization strategy based on frequency decomposition, so as to fuse low-frequency structural information that meets preset robust conditions and high-frequency details that meet preset reasonable conditions to obtain the second-stage optimized noise latent variables, and to obtain the external drawing results based on the second-stage optimized noise latent variables.

[0077] Optionally, in one embodiment of this application, the drawing module 300 includes an output unit.

[0078] The output unit is used to input the second-stage optimized noise latent variables into the pre-trained diffusion model to perform a complete reverse denoising process, so as to output the external drawing result containing the complete image content.

[0079] Optionally, in one embodiment of this application, the determining module 100 includes: an acquisition unit, a construction unit, and a first determining unit.

[0080] The acquisition unit is used to acquire the source image.

[0081] The construction unit is used to construct a binary mask corresponding to the source image.

[0082] The first determining unit is used to jointly encode the source image and the binary mask into a model input format to determine the region to be drawn.

[0083] Optionally, in one embodiment of this application, the optimization module 200 includes: a setting unit, a sampling unit, and a second determining unit.

[0084] The setting unit is used to set the number of time steps in the diffusion process.

[0085] The sampling unit is used to sample random noise tensors from a preset standard Gaussian distribution according to the number of time steps.

[0086] The second determining unit is used to determine the initial latent variables based on the noise tensor.

[0087] Optionally, in one embodiment of this application, the optimization module 200 includes: a first calculation unit, a second calculation unit, a third calculation unit, and an optimization unit.

[0088] The first computational unit is used to input the initial latent variables into the denoising network of the pre-trained diffusion model to extract the self-attention map of the denoising network, and to calculate the attention of the region to be drawn based on the self-attention map.

[0089] The second calculation unit is used to calculate the average attention score of the region to be drawn to itself and the average attention score of the region to be drawn to the source image region based on the attention of the region to be drawn and the binary mask.

[0090] The third calculation unit is used to calculate the attention alignment index based on the average attention score of the region to be drawn to itself and the average attention score of the region to be drawn to the source image region.

[0091] The optimization unit is used to minimize the attention alignment index as the optimization objective. It uses a preset gradient descent algorithm to iteratively update the mean and standard deviation of the Gaussian distribution followed by the initial input latent variables until the attention alignment index meets the preset convergence condition, thus obtaining the first-stage optimized noise latent variables.

[0092] Optionally, in one embodiment of this application, the drawing module 300 includes: a generation unit, a first extraction unit, a second extraction unit, an overlay unit, and an acquisition unit.

[0093] The generation unit is used to input the first-stage optimized noise latent variables into the pre-trained diffusion model, perform the reverse denoising process to generate preliminary external drawing image results, and apply the forward denoising process to the preliminary external drawing image results to generate an initial noise tensor containing low-frequency structural information.

[0094] The first extraction unit is used to apply a fast Fourier transform to the initial noise tensor and extract low-frequency structural information that meets the preset robustness conditions through a preset low-pass filter.

[0095] The second extraction unit is used to sample random variables from a preset standard Gaussian distribution, apply a fast Fourier transform, and extract high-frequency details that meet preset reasonable conditions through a preset high-pass filter.

[0096] The superposition unit is used to weight and superimpose low-frequency structural information that meets preset robustness conditions with high-frequency details that meet preset reasonable conditions to obtain a fused hybrid noise tensor.

[0097] The acquisition unit is used to apply the mixed noise tensor to the region to be drawn based on the binary mask, and retain the first-stage optimized noise latent variables of the source image region to obtain the second-stage optimized noise latent variables.

[0098] It should be noted that the foregoing explanation of the image outlining method based on attention alignment and frequency decomposition also applies to the image outlining device based on attention alignment and frequency decomposition in this embodiment, and will not be repeated here.

[0099] The image outlining device based on attention alignment and frequency decomposition proposed in this application adjusts the attention distribution between the region to be outlined and the source image region through attention alignment index to obtain the first-stage optimized noise latent variable. Then, based on the preset noise re-initialization strategy of frequency decomposition, it fuses low-frequency structural information and high-frequency details to obtain the second-stage optimized noise latent variable, which is input into the pre-trained diffusion model for image outlining. This device can finely optimize and reconstruct the initial input noise, guide the generation process, enable the pre-trained model to make full use of contextual information, improve semantic consistency, effectively suppress high-frequency artifacts, and enhance the visual realism of the outlined image, significantly improving realism, coherence, and visual aesthetics. At the same time, no training or fine-tuning of the pre-trained model is required, and it can be seamlessly integrated with various mainstream pre-trained models as a plug-and-play module, greatly reducing the application threshold.

[0100] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0101] When the processor 402 executes the program, it implements the image outlining method based on attention alignment and frequency decomposition provided in the above embodiments.

[0102] Furthermore, electronic devices also include: Communication interface 403 is used for communication between memory 401 and processor 402.

[0103] The memory 401 is used to store computer programs that can run on the processor 402.

[0104] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0105] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0106] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0107] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0108] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image outlining method based on attention alignment and frequency decomposition.

[0109] This application also provides a computer program product, including a computer program that can run computer instructions. When the computer instructions are executed by a processor, they implement the image outlining method based on attention alignment and frequency decomposition provided in this application.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0112] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0114] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0117] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An image outpainting method based on attention alignment and frequency decomposition, characterized in that, The method comprises the following steps: determining a region to be out-drawn according to a binary mask of a source image; determining an initial latent variable of a pre-trained diffusion model reverse denoising process, and iteratively optimizing the initial latent variable by using an attention alignment index to adjust the attention distribution between the region to be out-drawn and a source image region until the iteration ends, to obtain a first-stage optimized noise latent variable; processing the first-stage optimized noise latent variable by using a preset noise reinitialization strategy based on frequency decomposition to fuse low-frequency structural information satisfying a preset robust condition and high-frequency details satisfying a preset reasonable condition, to obtain a second-stage optimized noise latent variable, and obtaining an out-drawing result based on the second-stage optimized noise latent variable.

2. The method of claim 1, wherein, The out-drawing result based on the second-stage optimized noise latent variable comprises: inputting the second-stage optimized noise latent variable into the pre-trained diffusion model to perform a complete reverse denoising process to output the out-drawing result containing complete image content.

3. The method of claim 1, wherein, The region to be out-drawn determined according to the binary mask of the source image comprises: obtaining the source image; constructing a binary mask corresponding to the source image; jointly encoding the source image and the binary mask into a model input format to determine the region to be out-drawn.

4. The method of claim 1, wherein, The initial latent variable of the pre-trained diffusion model reverse denoising process comprises: setting the time step number of the diffusion process; sampling a random noise tensor from a preset standard Gaussian distribution according to the time step number; determining the initial latent variable according to the noise tensor.

5. The method of claim 1, wherein, The iteratively optimized initial latent variable by using the attention alignment index to adjust the attention distribution between the region to be out-drawn and the source image region until the iteration ends to obtain the first-stage optimized noise latent variable comprises: inputting the initial latent variable into the denoising network of the pre-trained diffusion model to extract a self-attention graph of the denoising network, and calculating the attention of the region to be out-drawn according to the self-attention graph; calculating the average attention score of the region to be out-drawn to itself and the average attention score of the region to be out-drawn to the source image region based on the attention of the region to be out-drawn and the binary mask, respectively; calculating the attention alignment index according to the average attention score of the region to be out-drawn to itself and the average attention score of the region to be out-drawn to the source image region; taking minimizing the attention alignment index as an optimization objective, iteratively updating the mean and standard deviation of the Gaussian distribution to which the initial input latent variable is subjected by using a preset gradient descent algorithm until the attention alignment index satisfies a preset convergence condition, to obtain the first-stage optimized noise latent variable.

6. The method of claim 1, wherein, The first-stage optimized noise latent variable is processed by using a preset noise reinitialization strategy based on frequency decomposition to fuse low-frequency structural information satisfying a preset robust condition and high-frequency details satisfying a preset reasonable condition to obtain a second-stage optimized noise latent variable, which comprises: inputting the first-stage optimized noise latent variable into the pre-trained diffusion model, performing a reverse denoising process to generate a preliminary outpainting image result, and applying a forward noise adding process to the preliminary outpainting image result to generate an initial noise tensor containing low-frequency structural information; applying a fast Fourier transform to the initial noise tensor and passing it through a pre-set low-pass filter to extract the low-frequency structural information satisfying the pre-set robust condition; sampling a random variable from a pre-set standard Gaussian distribution, applying a fast Fourier transform, and passing it through a pre-set high-pass filter to extract the high-frequency details satisfying the pre-set reasonable condition; weighting and superimposing the low-frequency structural information satisfying the pre-set robust condition and the high-frequency details satisfying the pre-set reasonable condition to obtain a fused hybrid noise tensor; applying the hybrid noise tensor to the outpainting region based on the binary mask and retaining the first-stage optimized noise latent variable of the source image region to obtain the second-stage optimized noise latent variable.

7. An image outliner based on attention alignment and frequency decomposition, comprising: comprising: a determination module configured to determine an outpainting region according to a binary mask of a source image; an optimization module configured to determine an initial latent variable of a reverse denoising process of a pre-trained diffusion model, and iteratively optimize the initial latent variable using an attention alignment indicator to adjust the attention distribution between the outpainting region and the source image region until the iteration ends, to obtain a first-stage optimized noise latent variable; an outpainting module configured to process the first-stage optimized noise latent variable using a pre-set noise reinitialization strategy based on frequency decomposition to fuse low-frequency structural information satisfying a pre-set robust condition and high-frequency details satisfying a pre-set reasonable condition, to obtain a second-stage optimized noise latent variable, and to obtain an outpainting result based on the second-stage optimized noise latent variable.

8. An electronic device, comprising: comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the image outpainting method based on attention alignment and frequency decomposition according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the image outpainting method based on attention alignment and frequency decomposition according to any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the image outpainting method based on attention alignment and frequency decomposition according to any one of claims 1-6.

Images