Image editing method and device and storage medium

Abstract

The invention discloses an image editing method and device and a storage medium. The method comprises the steps that diffusion processing is conducted on an original image through a diffusion model; determining the total number of noise reduction iterations by using a first model trained in advance; and in each noise reduction processing process, sampling a network module of the noise prediction network in the diffusion model by using a pre-trained second model, and carrying out noise reduction processing by using the network module selected by sampling to obtain an edited target image. By applying the scheme of the embodiment of the invention, the first model determines the total number of iterations of noise reduction, and the second model samples the network module of the noise prediction network, so that the proper number of iterations and the proper network module can be adaptively selected for different images, and the noise reduction efficiency is improved on the basis of ensuring the image quality. The complexity of a long iteration process and a network structure is greatly avoided, and the generation efficiency of image editing is effectively improved.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to an image editing method, an image editing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Image editing technology is one of the important applications of machine vision. Image editing refers to the operations of repairing, modifying, compositing, optimizing, or creating new visual effects on images. To improve image editing and make the edited images more natural and realistic, large-scale data acquisition and complex model structures, such as diffusion models, are frequently used in recent years. A diffusion model is a generative model that generates complex data distributions. Its core idea is to simulate a diffusion process, gradually adding noise to the original data to convert it into a noisy image, and then learning a denoising process to recover the original data from the noise. The diffusion process is the forward process, while the denoising process is the reverse process. While diffusion models produce good generative results, their low efficiency due to the complexity of the model's network structure and the lengthy iteration process hinders their application in image editing. Summary of the Invention

[0003] In view of the above-mentioned prior art, this invention discloses an image editing method that can overcome the shortcomings of complex model network structure and long iteration process, thereby improving generation efficiency.

[0004] This application provides an image editing method including:

[0005] The original image is diffused using a diffusion model;

[0006] The mask image and the occluded image are used as inputs to the first model, and the total number of noise reduction iterations is determined using the pre-trained first model.

[0007] The diffusion model performs noise reduction on the diffused image according to the number of noise reduction iterations. In each noise reduction process, the network module of the noise prediction network in the diffusion model is sampled using a pre-trained second model. The network module selected by the sampling calculates the noise reduction and obtains the edited target image.

[0008] In view of the above-mentioned prior art, this invention also discloses an image editing device that can overcome the defects of complex model network structure and long iteration process, thereby improving generation efficiency.

[0009] This application provides an image editing device comprising:

[0010] The diffusion model module is used to perform diffusion processing on the original image, and then to perform noise reduction processing on the diffused image according to the number of noise reduction iterations to obtain the edited target image.

[0011] The first model module is used to determine the total number of noise reduction iterations based on the mask image and the occluded image;

[0012] The second model module is used to sample the network modules of the noise prediction network in the diffusion model during each noise reduction process, and to calculate the noise reduction process by the network modules selected by the sampling.

[0013] In view of the above-mentioned prior art, embodiments of the present invention also disclose a computer-readable storage medium that can overcome the defects of complex model network structure and long iteration process, thereby achieving the goal of improving generation efficiency.

[0014] This application provides a computer-readable storage medium comprising: computer instructions stored thereon, wherein when executed by a processor, the instructions can implement the steps of any of the above-described image editing methods.

[0015] In view of the above-mentioned prior art, this invention also discloses a computer program product that can overcome the defects of complex model network structure and long iteration process, thereby improving generation efficiency.

[0016] This application provides a computer program product including computer instructions that, when executed by a processor, implement the image editing method as described in any of the preceding claims.

[0017] In summary, this application addresses the shortcomings of existing diffusion models, such as complex network structures and lengthy iteration processes, by adding a first model and a second model. The first model determines the total number of denoising iterations, while the second model samples the network modules in the diffusion model. The selected network modules then perform denoising calculations. This allows for adaptive selection of different iteration numbers and network modules for different images, ensuring image quality while significantly improving the efficiency of image editing. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a logical structure diagram of the diffusion model in Embodiment 1 of the method of this application.

[0020] Figure 2 This is an example of how existing technologies use diffusion models to process images.

[0021] Figure 3 This is a flowchart of an embodiment of the image editing method of this application.

[0022] Figure 4 This is an example of how different mask morphological features affect diffusion model processing in Embodiment 1 of this application.

[0023] Figure 5 This is a flowchart of the diffusion treatment stage in Embodiment 2 of this application.

[0024] Figure 6 This is a flowchart of the noise reduction process in Embodiment 3 of this application.

[0025] Figure 7 This is a flowchart of the method implemented by the first model in the fourth embodiment of the method of this application.

[0026] Figure 8 This is a schematic diagram of the network layer structure of the first model in Embodiment 4 of this application.

[0027] Figure 9 This is a flowchart of the method implemented by the second model in the fifth embodiment of the method of this application.

[0028] Figure 10 This is a schematic diagram of the network layer structure of the second model in Embodiment 5 of the method of this application.

[0029] Figure 11 This is an example of the sampling results of the network module of the noise prediction network in Embodiment 5 of this application.

[0030] Figure 12 This is a logical structure diagram of the diffusion model in Embodiment Six of the present application.

[0031] Figure 13 This is a flowchart of Embodiment Six of the Image Editing Method of this Application.

[0032] Figure 14 This is a schematic diagram illustrating the effect of replacing a specific area in Embodiment 7 of this application.

[0033] Figure 15 This is a schematic diagram illustrating the effect of clearing a specific area using the method embodiment eight of this application.

[0034] Figure 16 This is a schematic diagram illustrating the effect of drawing a specific area in Embodiment 9 of the present application.

[0035] Figure 17 This is a schematic diagram illustrating the effect of modifying a specific area in Embodiment 10 of this application.

[0036] Figure 18 This is an embodiment of the device for image editing in this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0040] To address the shortcomings of existing diffusion models, such as complex network structures and lengthy iteration processes, this application proposes a method that involves sampling the network modules of the noise prediction network for each iteration, with only the sampled network modules participating in the noise reduction calculation. This significantly reduces the iteration process of the diffusion model and effectively improves image editing.

[0041] Figure 1 This is a logical structure diagram of the diffusion model in Embodiment 1 of the method of this application. For example... Figure 1 As shown, the model includes a diffusion process 101, a noise reduction process 102, a first model 103, and a second model 104. The diffusion process 101 and the noise reduction process 102 are similar to the diffusion models in the prior art. The difference is that the embodiments of this application add a first model 103 and a second model 104.

[0042] Diffusion models can be implemented using deep neural network architectures, such as U-Net or Transformer. The diffusion process involves progressively adding noise to the original image, transforming it into a noisy image with a standard normal distribution. Assuming an original image is denoted as x0, it is gradually processed through a Gaussian encoder with T noise additions to obtain x1, x2, x3, ..., x... T The final x T It follows a standard normal distribution. The core idea of ​​the diffusion process can be represented as:

[0043]

[0044] Wherein, the distribution of the q diffusion process, q(x) t |x t-1 ) indicates that x is known t-1 time x t The distribution of , where t is any time from 1 to T, and α is the hyperparameter of the Gaussian encoder. x represents t Follows a normal distribution. and (1-α) t I is its parameter, where I represents the identity matrix.

[0045] Conversely, the noise reduction process is represented as p θ (x t-1 |x t This involves progressively denoising and restoring a noisy image, and transforming this restoration process into training a noise prediction network to estimate x. t The noise in the network is analyzed, and the L2 distance between the predicted noise and the actual noise is minimized to optimize the network parameters θ. Once the noisy network is trained, the diffusion model can utilize the learned p... θ (x t-1 |x t For random noise x T Perform T denoising operations to generate a clean original sample x0. Figure 2 This is an example of how existing techniques utilize diffusion models to process images. For example... Figure 2 As shown, an original image x0 of a puppy is processed by diffusion q(x t |x t-1 Step by step, we obtain x1, x2, x3, ..., x T x T The image x is transformed into a pure noise image that follows a standard normal distribution. T Conversely, the purely noisy image x T Through noise reduction processing p θ (x t-1 |x t x is obtained step by step T-1 ...x2,x1,...x0, to obtain a clear image.

[0046] This application's embodiments are implemented based on the aforementioned diffusion model. It should be noted that this application's embodiments pertain to image editing operations, including repair, modification, compositing, optimization, or creating new visual effects. These operations only apply to the portion of the image that needs editing; other areas remain unchanged, but the diffusion processing method for the portion requiring editing is the same as described above.

[0047] Figure 3 This is a flowchart of an embodiment of the image editing method of this application. Figure 3 As shown, the method includes:

[0048] Step 301: Perform diffusion processing on the original image using a diffusion model.

[0049] Step 302: Use the mask image and the occlusion image as input to the first model, and use the pre-trained first model to determine the total number of noise reduction iterations.

[0050] As mentioned above, the diffusion model is divided into a diffusion process and a denoising process. Step 301 in Embodiment 1 of this application is a step implemented in the diffusion process, while step 303 below is a step implemented in the denoising process.

[0051] Existing diffusion models typically require hundreds or even thousands of iterations to generate a final high-quality image. Each iteration consumes time for computation and updates, significantly reducing generation efficiency. To improve efficiency, step 302 can predict the total number of denoising iterations (i.e., denoising steps) for different images, instead of having the diffusion model iterate automatically. The prediction of the total number of denoising iterations is performed by the first model 103. In practical applications, the first model 103 can be implemented using machine learning, as long as the corresponding total number of denoising iterations can be determined for the input original image.

[0052] In image editing operations, the inventors of this application discovered that the number of denoising iterations is affected by various factors, such as the complexity of the original image background and the mask morphology features. Specifically, if the original image background is complex, including a large amount of detail and texture, the diffusion model will need to spend more computational resources to understand its semantics to ensure that the generated content is consistent with the style of the original image background region. Mask morphology features can provide different levels of guidance; a mask with a complete morphology can provide the diffusion model with clearer editing guidance, effectively alleviating the editing burden. In addition, the diffusion model can be implemented using different specific models, and different models, due to their own structure and design characteristics, will exhibit different computational power allocation preferences, and the model size and hyperparameter settings will also affect the generation process.

[0053] In view of the above findings, this application's embodiments creatively use the mask image and occlusion image as inputs to the first model 103. The first model 103 determines the total number of denoising iterations by referring to these factors. The mask image refers to the image that segments the original image into parts that need to be edited and other parts that do not need editing. The occlusion image refers to the original image with the parts that need to be edited removed. In another embodiment, to more accurately determine the total number of denoising iterations, a guiding scale can also be used as input to the first model 103. The guiding scale is a hyperparameter of the diffusion model, used to balance the randomness and determinism of the model during the generation process. The larger the guiding scale, the less computational power the model requires; conversely, the smaller the guiding scale, the more computational power the model requires. That is, the mask image corresponds to the aforementioned mask morphology features, the occlusion image corresponds to the aforementioned complexity of the original image background, and the guiding scale corresponds to the aforementioned model preferences. This application's embodiments can input the mask image, occlusion image, and guiding scale into the first model, so that the output total number of denoising iterations fully considers factors such as the complexity of the original image background, mask morphology features, and model preferences.

[0054] Assumption Figure 4 This is an example of how different mask morphological features affect diffusion model processing in Embodiment 1 of this application. For example... Figure 4 As shown, the top image is of a person holding a monkey, and the bottom image is of a snake crawling on the ground. In the top image, the monkey's mask image is incomplete due to the person holding it, making accurate identification based on its shape impossible. In the bottom image, the snake's mask image is complete, allowing for accurate identification based on its shape. Guided by these two different mask morphological features... Figure 4 The results of iterations up to the 4th and 8th iterations are also shown. From the examples above, it can be seen that the editing effect of the monkey in the top image is not very good, while the editing effect of the snake in the bottom image is more realistic. Furthermore, assuming the model is required to generate a red cat, if the guidance scale is large and detailed text prompts are provided, the model will most likely generate a red cat image, and each image will be quite similar. However, if the guidance scale is small, the model may play it by itself, potentially producing a red dog or a blue cat, with very low similarity between images. As can be seen from the examples above, this application embodiment considers the mask image, occlusion image, and guidance scale to be factors affecting the total number of noise reduction iterations, especially the mask image and occlusion image, and therefore uses them as input to the first model 103.

[0055] Step 303: The diffusion model performs noise reduction on the image after diffusion processing according to the number of noise reduction iterations. In each noise reduction process, the network module of the noise prediction network in the diffusion model is sampled using the pre-trained second model. The selected network module is used to calculate the noise reduction and obtain the edited target image.

[0056] Existing diffusion models have very complex network structures, also known as noise prediction networks, typically composed of numerous network modules. The inventors of this application have discovered that the importance of these network modules varies at different generation stages. In diffusion models, the computational resources required for each iteration depend on the previous prediction state and the model's own inference stage. For example, if the prediction result at the previous time step has effectively approximated the original distribution, the computational power required to approximate the original distribution at the current time step can be reduced accordingly; conversely, the computational power increases. Furthermore, early iterations focus on restoring the coarse-grained structure of the image, requiring relatively fewer computational resources, while later iterations focus on refining the image, such as adding more texture, color, and details, requiring more computational power for fine-grained noise estimation. Therefore, step 303 of this embodiment adaptively selects different network modules for each generation step, avoiding the need to use all network modules in every iteration, thus optimizing resources and improving generation efficiency. In this embodiment, the sampling of network modules is implemented by the second model 104. In practical applications, the second model 104 can be implemented using machine learning, as long as the network modules of the noise prediction network can be sampled to reflect the importance of each network module at different stages.

[0057] By applying the scheme of this application, since a first model and a second model are added to the diffusion model, the first model determines the total number of denoising iterations, and the second model samples the network modules of the noise prediction network. Therefore, when editing images, the number of iterations and the number of network modules involved in the denoising calculation can be reduced for different situations. In other words, the embodiments of this application can adaptively select appropriate iterations and appropriate network modules for different images, thereby greatly avoiding lengthy iteration processes and complex network structures while ensuring image quality, and effectively improving the generation efficiency of image editing.

[0058] Figure 5 This is a flowchart of the diffusion process in Embodiment 2 of this application. For example... Figure 5 As shown, this method includes:

[0059] Step 501: Encode the original image to obtain the latent features of the original image, and map the original image from the pixel space to the latent space.

[0060] In machine learning, latent space is a low-dimensional, continuous representation space with semantic structure that exists after compressing and abstracting complex, high-dimensional data. In the embodiments of this application, image processing is performed in the latent space, and the processed data is finally returned to the pixel space.

[0061] Step 502: Encode the occluded image to obtain the latent features of the occluded image, and map the occluded image from the pixel space to the latent space.

[0062] The diffusion model processes images in the latent space, so both the original image and the occluded image need to be encoded and mapped into the latent space.

[0063] Step 503: Add noise to the latent features of the original image to obtain noisy latent features that conform to a standard normal distribution.

[0064] Step 504: Concatenate the latent features of the noisy image, the latent features of the occluded image, and the mask image to obtain the initial latent features of the noisy image.

[0065] Here, the mask image does not require latent space mapping; only interpolation is needed. Thus, the diffusion model obtains the initial noisy image latent features obtained by concatenating the latent features of the noisy image, the occluded image, and the mask image. This data includes noisy image information conforming to a standard normal distribution, mask image information for segmenting the regions to be edited and those not to be edited, and the original image information of the regions not to be edited. This facilitates subsequent editing of only the mask portion while preserving the original image information of the occluded portion.

[0066] It should be noted that, while performing the diffusion processing stage described above, the embodiments of this application can also utilize the first model to determine the total number of denoising iterations. This operation of determining the total number of denoising iterations can be performed in parallel with the diffusion processing, as long as it occurs before the denoising processing.

[0067] Figure 6 This is a flowchart of the noise reduction process in Embodiment 3 of this application. Figure 6 As shown, the method includes:

[0068] Step 601: Use the initial noisy image latent features as input to the current noisy image latent features of the noise prediction network.

[0069] Step 602: Use a noise prediction network to perform noise prediction processing, determine the noise data features of this iteration, and then perform noise reduction processing on the latent features of the current noise image based on the noise data features of this iteration to obtain the latent features of the noise image after noise reduction.

[0070] Step 603: Use the latent features of the noise image after this denoising as the new latent features of the current noise image, return to step 602, and continue until the total number of denoising iterations is completed to obtain the final latent features of the noise image.

[0071] The above steps 602 and 603 are iterative processes, and the number of iterations is the total number of denoising iterations. The noise prediction network has a complex structure, typically comprising several network modules, each with a different resolution achieved through sampling operations. In this embodiment, the network modules are divided according to different resolutions, with each module processing feature data corresponding to its resolution. Each network module has a similar internal structure, consisting of convolutions, sampling, and skip connections. In practical applications, the iterative denoising process also includes other steps, such as sampling or inputting conditions that meet the editing objectives; details can be found in existing diffusion models, which will not be elaborated here.

[0072] Step 604: Decode the latent features of the final noisy image and map them back from the latent space to the pixel space to obtain the edited target image. Since the above noise reduction process is performed in the latent space, it is necessary to map it back to the pixel space after the noise reduction process is completed to obtain an image visible to the human eye.

[0073] At this point, the obtained target image is an edited image that does not contain noise. Because this embodiment edits the masked area—such as modifying, repairing, compositing, and optimizing—the displayed image of the edited masked area may differ from the original image. Figure 4 As can be seen, the snake's texture is different before and after editing, but the generated result is very realistic and natural, and it can be naturally connected with the occluded image as the background.

[0074] It is important to note that before each iteration in steps 602 and 603, this application further samples the network modules of the noise prediction network using the second model, and determines which network modules participate in the noise reduction calculation based on the sampling results. That is, in each iteration, not all network modules in step 602 perform noise reduction on the current noisy image latent features; only the sampled and selected network modules participate in the calculation. In this embodiment, the sampled and selected modules can be referred to as "activated," while those not sampled and selected are referred to as "frozen." In practical applications, activated network modules participate in the calculation, and frozen network modules can be replaced by bilinear interpolation to obtain the output features, i.e., the noise reduction image latent features.

[0075] Because diffusion models involve lengthy iterations, existing technologies employ various methods to improve efficiency. For example, stochastic differential equations (SDEs) or ordinary differential equations (ODEs) are used to model the generation process, resulting in a sampler that requires no training and compensating for quality loss caused by reduced time steps. Knowledge distillation or learning fast samplers are also used to accelerate the diffusion model. However, these methods lack flexibility, applying the same number of steps to all samples. The first model provided in this application differs from the aforementioned existing technologies, being implemented using machine learning methods such as simple stacked neural networks. To better illustrate the solution of this application, the following method embodiment four describes a specific implementation. Figure 7 This is a flowchart illustrating the method implementation of the first model in Embodiment 4 of this application. For example... Figure 7 As shown, the method includes:

[0076] Step 701: Use the mask image, occlusion image, and guide scale as input to the first model.

[0077] Step 702: Perform calculations using the network layers of the first model, which include a first convolutional layer, a first pooling layer, a first linear layer, and a first activation function.

[0078] The network layers of the first model here can be simple stacked network layers, including convolutional layers, pooling layers, linear layers, and activation functions. To distinguish them from other models in this application, they are referred to here as the first convolutional layer, the first pooling layer, the first linear layer, and the first activation function, respectively. Figure 8 This is a schematic diagram of the network layer structure of the first model in Embodiment 4 of this application. For example... Figure 8 As shown, the network layer consists of two 3x3 convolutional layers, one 512x50 linear layer, one 64x64 pooling layer, and a sigmoid activation function. Each layer's data dimensions include the number of samples, feature dimension, width, and height. The convolutional layer has a data dimension of (1, 512, 64, 64), the linear layer has a data dimension of (1, 50, 64, 64), the pooling layer has a data dimension of (1, 50, 1, 1), and the activation function has a data dimension of (1, 50). In other words, the mask image, occlusion image, and guiding scale are first fused into features of size (1, 512, 64, 64). These features are then passed sequentially through two 3x3 convolutional layers, followed by a 512x50 linear layer to reduce the feature dimension, then a 64x64 pooling layer to reduce the resolution, and finally, a sigmoid activation function is applied to obtain several probability values. The highest probability value is selected as the total number of denoising iterations.

[0079] This section only introduces an example of the specific implementation of the first model. In practical applications, other forms of network layers can also be used, and it is not limited to this, as long as the total number of noise reduction iterations can be determined based on the mask image, the occluded image, and the guiding scale.

[0080] Step 703: Determine the total number of noise reduction iterations based on the output of the first model.

[0081] Using the method described in Embodiment 4 above, the total number of denoising iterations for the image to be edited can be determined based on a diffusion model. Those skilled in the art will understand that when image editing is relatively simple, the total number of denoising iterations is correspondingly fewer; when image editing is relatively complex, the total number of denoising iterations is correspondingly more.

[0082] Due to the complexity of diffusion models, existing technologies employ other methods to improve efficiency. For example, the noise-reducing diffusion-aware model reduces the overall computational load of the diffusion generation process by adaptively pruning the U-net to assess the importance of each generation step. Another example is the console diffusion transformer (DyDiT), which dynamically adjusts computational power along the temporal and spatial dimensions during generation. However, these methods do not address the case of editing only a portion of the image, nor do they consider the impact of mask and occlusion images on the generation process. In practical applications, the second model in the above embodiments is implemented using machine learning, such as a lightweight neural network. To better illustrate the solution of this application, the following method embodiment five describes a specific implementation. Figure 9 This is a flowchart illustrating the method implementation of the second model in Embodiment 5 of this application. For example... Figure 9 As shown, the method includes:

[0083] Step 901: The noise prediction network is pre-divided into multiple network modules according to the resolution change.

[0084] As mentioned earlier, noise prediction networks have complex structures. In this embodiment, they are divided into different network modules according to different resolutions, and each network module processes feature data at a corresponding resolution. Each network module has a similar internal structure, consisting of convolution, sampling, skip connections, and other components.

[0085] Step 902: Use the current noise image, total number of denoising iterations, current number of denoising iterations, and mask image of the noise prediction network as input to the second model.

[0086] Step 903: Calculate using the network layer of the second model to obtain the calculation result of the network layer of the second model. The network layer of the second model includes a second convolutional layer, a second pooling layer, a second linear layer, and a second activation function.

[0087] Step 904: Determine the probability of each network module in the noise prediction network being selected based on the calculation results of the network layers of the second model.

[0088] Step 905: Determine the sampling results based on the probability of the network module being selected.

[0089] Steps 902 to 905 above describe the process of sampling network modules in the noise prediction network. The inventors of this application have discovered that factors affecting the sampling results include at least the current noise image, the mask image, the total number of denoising iterations, and the current number of denoising iterations. Here, the current noise image and the mask image represent the image information that the noise prediction network needs to process; the total number of denoising iterations represents the total number of target iterations that the noise prediction network needs to process; and the current number of denoising iterations represents the number of denoising iterations that the noise prediction network is currently performing. In other words, by referring to the above factors, the second model can determine which network modules in the noise prediction network are activated and which are frozen, thus achieving the sampling objective.

[0090] In practical applications, the network layers of the second model adopt a lightweight design, including convolutional layers, pooling layers, linear layers, and activation functions. To distinguish them from other models in this application, they are referred to here as the second convolutional layer, the second pooling layer, the second linear layer, and the second activation function, respectively.

[0091] Figure 10 This is a schematic diagram of the network layer structure of the second model in Embodiment 5 of the method of this application. For example... Figure 10 As shown, the network layer includes: three 3*3 convolutional layers, one 64*64 pooling layer, one 64*8 linear layer, and an activation function sigmoid. The data dimensions of each layer include the number of samples, feature dimension, width, and height. Specifically, the data dimensions of the first two convolutional layers are (1, 256, 64, 64), the data dimensions of the last convolutional layer are (1, 256, 64, 64), the data dimensions of the linear layer are (1, 8), and the data dimensions of the activation function are (1, 8). The method for calculating using the network layer of the second model in step 903 includes: calculating the current noisy image using the second convolutional layer to obtain a temporary convolution result; fusing the temporary convolution result with the total number of denoising iterations, the current number of denoising iterations, and the mask image to obtain a fused result; and calculating the fused result using the second convolutional layer, the second pooling layer, the second linear layer, and the second activation function to obtain the calculation result of the network layer of the second model. In other words, the current noisy image is first passed through two 3x3 convolutional layers to obtain a temporary convolution result. This temporary convolution result is then fused with the total number of denoising iterations, the current number of denoising iterations, and the mask image to obtain a fused result. This fused result is then passed through another 3x3 convolutional layer, followed by a 64x64 pooling layer to reduce resolution, a 64x8 linear layer to reduce feature dimensions, and finally a sigmoid activation function to obtain several probability values. In practical applications, a threshold can be set for the probability values, with values ​​greater than the threshold assigned as 1 and values ​​less than the threshold assigned as 0. Therefore, 1 corresponds to activating the corresponding network module, and 0 corresponds to freezing the corresponding network module.

[0092] Figure 11This is an example of the sampling results of the network module of the noise prediction network in Embodiment 5 of this application. For example... Figure 11 As shown, the noise prediction network includes several network modules. Solid boxes represent selected network modules for sampling, and dashed boxes represent unselected network modules. That is, when the noise prediction network is used for noise prediction processing in step 602 of the method embodiment three, only the network modules represented by solid boxes participate in the calculation, and the network modules represented by dashed boxes can be replaced by bilinear interpolation. Finally, the output features are obtained, which are the latent features of the denoised noise image.

[0093] As mentioned earlier, existing diffusion models can be implemented using deep neural network architectures such as U-Net or Transformer. Embodiment Six of this application uses U-Net as an example to specifically illustrate the technical solution. In this embodiment, image editing includes two core tasks: generation and reconstruction. It requires not only the ability to generate clear image content from the masked area, but also the preservation of image information from other unmasked areas during the generation process, and a seamless and natural transition between the newly generated image content and the preserved image content.

[0094] Figure 12 This is a logical structure diagram of the diffusion model in Embodiment Six of this application. For example... Figure 12 As shown, the model includes an encoder 1201, a noise addition and feature fusion section 1202, a U-Net network denoising section 1203, a sampler 1204, a decoder 1205, a first model 1206, and a second model 1207. Specifically, the encoder 1201 and the noise addition and feature fusion section 1202 correspond to the diffusion process 101 in Method Embodiment 1; the U-Net network denoising section 1203, the sampler 1204, and the decoder 1205 correspond to the denoising process 102 in Method Embodiment 1; the first model 1206 is the first model 103 in Method Embodiment 1; and the second model 1207 is the second model 104 in Method Embodiment 1.

[0095] Figure 13 This is a flowchart of Embodiment Six of the Image Editing Method of this Application. Figure 13 As shown, the method includes:

[0096] Step 1301: Encode the original image to obtain the latent features of the original image, and map the original image from the pixel space to the latent space.

[0097] Step 1302: Encode the occluded image to obtain the latent features of the occluded image, and map the occluded image from the pixel space to the latent space.

[0098] Steps 1301 and 1302 above involve encoding the original image and the occluded image respectively and mapping them to the latent space. Figure 12In the image, the original image is a bird with a white belly, and there are corresponding mask and occluded images. The original image and the occluded image will be mapped to the latent space respectively. For ease of observation, Figure 12 Images in the pixel space and latent space are not distinguished and are displayed as visible images. However, it is important to note that the image mapped to the latent space after encoding has generated latent features, rather than being the displayed pixel image.

[0099] Step 1303: Add noise to the latent features of the original image to obtain noisy latent features that conform to a standard normal distribution.

[0100] exist Figure 12 In the diagram, we can see that the latent features of the original image are transformed into the latent features of a noisy image by adding noise.

[0101] Step 1304: The noisy image latent features, the occluded image latent features, and the mask image are stitched together to obtain the initial noisy image latent features.

[0102] In practical applications, both the original image and the occluded image are encoded and mapped to latent features, while the mask image does not need to be encoded and can be interpolated. Then, the three images are concatenated along the feature dimension. For example, if the latent feature x is represented as (b, c, h, w) and the latent feature y is represented as (b, c, h, w), then the concatenated image can be represented as (b, 2c, h, w).

[0103] Step 1305: Determine the total number of noise reduction iterations using the pre-trained first model based on the mask image, occlusion image, and guiding scale.

[0104] Steps 1301 to 1304 described above constitute a diffusion process, equivalent to step 301 in Method Embodiment 1 or steps 501 to 504 in Method Embodiment 2. In this embodiment, the total number of denoising iterations needs to be determined using the first model in step 1305 to determine the number of iterations for subsequent steps 1307 to 1308. There is no strict execution order between steps 1305 and steps 1301 to 1304, as long as step 1305 occurs before the subsequent denoising process. Specific implementations of steps 1301 to 1304 can be found in Method Embodiment 2, and specific implementations of step 1305 can be found in Method Embodiment 4; these will not be repeated here.

[0105] Step 1306: Use the initial noisy image latent features as input to the current noisy image latent features of the noise prediction network.

[0106] Step 1307: Sample the network modules of the U-Net network using the pre-trained second model.

[0107] Step 1308: The network module selected by sampling in the U-Net network performs noise prediction processing to determine the noise data features of this iteration. Then, based on the noise data features of this iteration, the latent features of the current noise image are denoised to obtain the latent features of the noise image after denoising.

[0108] Step 1309: Use the noise image latent features after this denoising as the new current noise image latent features, return to step 1307, and continue until the total number of denoising iterations is completed to obtain the final noise image latent features.

[0109] Step 1310: Decode the latent features of the final noisy image and map them back from the latent space to the pixel space to obtain the edited target image.

[0110] Steps 1306 and 1308 to 1310 above are noise reduction processes, equivalent to step 302 in Method Embodiment 1, or steps 601 to 604 in Method Embodiment 3. In this embodiment, before each noise reduction process of the U-Net network, step 1307 is used to sample the network modules of the U-Net network to determine the network modules participating in the noise reduction calculation. The specific implementation of steps 1306 and 1308 to 1310 can be found in Method Embodiment 3, and the specific implementation of step 1307 can be found in Method Embodiment 5. Furthermore, this embodiment uses the U-Net network as an example; in practical applications, other networks such as Transformer can also be used. Assuming that steps 1307 to 1309 iterate k times to obtain the denoised x0, the decoder will then finally produce a clear, edited target image. It can be seen that the target image here is different from the original image; the bird's belly is edited to black, and there is a white streak on the wings, but the background image (i.e., the occluded image) remains unchanged, and the edited bird blends naturally and realistically with the background.

[0111] Those skilled in the art will understand that the designed model needs to be trained before it can be used. Another embodiment of this application proposes an end-to-end training method for the improved diffusion model described above. The loss function for end-to-end training proposed in this embodiment includes two parts: one is the loss of the diffusion model itself, referred to here as the first loss L. SD Secondly, there is the loss introduced by the first and second models mentioned above, which is referred to here as the second loss L. FLOPs The total loss function for end-to-end training in this embodiment is expressed as L. Total =λ1L SD +λ2L FLOPs Where λ1 and λ2 are hyperparameters used for weighting, and the second loss is... The F dyanmic F represents the computational cost of the first and second models involved in the calculation.static This indicates the computational cost involved in the diffusion model itself.

[0112] In other words, the computational complexity F involved in the calculation is calculated based on the total number of noise reduction iterations determined by the first model and the network size determined by the network sampling of the second model. dyanmic Generally, the more total denoising iterations and the more network modules activated by network sampling, the higher the computational cost F. dyanmic The larger it is, the lower its efficiency. Second loss. This can be considered the ratio of the computational effort involved by the first and second models to that of the diffusion model itself. A smaller ratio means less computational effort from the first and second models in the improved part, resulting in higher efficiency, but also a greater degree of image quality degradation. Conversely, a larger ratio means more computational effort from the first and second models in the improved part, resulting in lower efficiency, but also higher image quality accuracy.

[0113] Therefore, in order to balance image accuracy and editing efficiency, the embodiments of this application retain the loss function L of the diffusion model itself. SD This is to ensure that the image quality is not severely compromised. Thus, the first loss L... SD Second loss L FLOPs The weights λ1 and λ2 are adjusted to balance image accuracy and efficiency. A larger λ1 emphasizes image precision, while a larger λ2 emphasizes efficiency. In practical applications, adjusting λ1 and λ2 appropriately achieves both image precision and model efficiency.

[0114] In practical applications, the above-described technical solutions of this application can be applied to a variety of different scenarios. Several specific embodiments are described below.

[0115] In method embodiment seven, it is assumed that a specific area of ​​an image on a mobile phone needs to be replaced. For example, changing the color of a person's clothes, changing the style or texture of an item, or changing the background scene. Figure 14 This is a schematic diagram illustrating the effect of replacing a specific area in Embodiment 7 of this application. Figure 14 As shown, the first row contains the original images: a steamed bun and condiments, a dolphin in the water, a young man wearing a hat, a pair of boots, a car, and a flatbread. The second row contains the mask images corresponding to specific areas. The third row contains the target images after the specific areas have been replaced. We can see that in the first image, the color of the condiments has changed; in the second image, the lighting and texture on the dolphin have changed; in the third image, the texture of the hat has changed; in the fourth image, the style of one of the boots has changed; in the fifth image, both the color and style of the car have changed; and in the sixth image, the flatbread has become a cake, and the covering has also changed.

[0116] In practical implementation, the embodiments of this application can allocate different computing power based on the specific areas in the original image that need to be replaced. For example, in the first image, only the color of the seasoning is adjusted, the mask area is small and relatively simple, so less computing power can be allocated, meaning fewer total noise reduction iterations and fewer network modules activated by network sampling. In the second image, the color and lighting of the dolphin are adjusted, the mask area is larger, so slightly more computing power can be allocated, meaning more total noise reduction iterations and more network modules activated by network sampling. This continues, gradually increasing the allocated computing power based on the specific areas to be replaced. In the last image, the mask image area for the pie is the largest, and the generated cake texture is very complex, therefore the most computing power will be allocated. In other words, the embodiments of this application, using the aforementioned technical solution, can allocate different computing power based on task complexity, maximizing generation efficiency while maintaining image quality.

[0117] In method embodiment eight, suppose it is necessary to enhance the image on the mobile phone by deleting elements in a specific area. For example, deleting blemishes and clutter, removing unwanted objects such as telephone poles, passersby, and garbage, and deleting watermarks, text marks, or image icons. Figure 15 This is a schematic diagram illustrating the effect of cleaning a specific area using the method embodiment eight of this application. For example... Figure 15 As shown, the first row contains the original images: a forest scene, indoor scene one, a bag of rice, a close-up of clothing, and indoor scene two. The second row corresponds to the mask images for specific areas. The third row corresponds to the target images after replacing the specific areas. It can be seen that in the first image, the pedestrians in the forest were removed; in the second image, the horizontal sofa was removed; in the third image, the text near the rice was removed; in the fourth image, the text near the suit was removed; and in the fifth image, the furniture was removed. It can also be seen that the removed areas were further repaired to blend naturally with the rest of the image.

[0118] In practical implementation, the embodiments of this application can also allocate different computing power according to the specific areas in the original image that need to be replaced. For example, the mask images of pedestrians in the first image and the mask images of the sofa in the second image are relatively regular and easy to remove. Moreover, the specific area can be well repaired based on the surrounding environment analysis. Therefore, allocating less computing power to them can yield a target image of good quality. Conversely, the mask images of the text around the objects in the third and fourth images, and the furniture in the fifth image, are relatively fragmented, and more computing power can be allocated to them. The embodiments of this application utilize the technical solution described above to allocate different computing power according to the task complexity, thereby maximizing the generation efficiency while meeting image quality requirements.

[0119] In Method Implementation Example 9, it is assumed that an image needs to be externalized for use in digital art creation and photography. Externalization refers to creatively extending and expanding upon an existing image. For example, artists break the boundaries of traditional images to create unique and imaginative works of art. Photographers use externalization to broaden their field of vision and increase the visual impact of their photographs. In virtual reality (VR) and augmented reality (AR) applications, image externalization can generate more realistic and immersive virtual scenes, enhancing the user experience. Figure 16 This is a schematic diagram illustrating the effect of drawing a specific area using the method embodiment nine of this application. For example... Figure 16 As shown, the first row represents the original images: a snow-capped mountain and lake scene, a traditional Chinese ink painting scene, and a landscape created through digital art. The original images are relatively narrow, with the surrounding black areas representing specific regions. The second row corresponds to the target images where the original images are drawn onto specific regions. It can be seen that the drawn areas expand the original images, providing a wider field of view, and the drawn portion blends well with the original image. Similarly, in practical implementation, the embodiments of this application can allocate different computing power based on the specific regions to be drawn, maximizing generation efficiency while maintaining image quality.

[0120] In method embodiment ten, it is assumed that video needs to be repaired to restore damaged areas. Video is a common multimedia type on the Internet, but bitstream corruption due to compression, storage, and transmission processes is a frequent problem. Different types of corruption factors can cause varying degrees of damage or error patterns in the decoded frames of the video. This application embodiment can utilize the technical solutions in the above method embodiments to determine a mask image based on the damaged area, use the surrounding space and adjacent frame content as references, and fill the damaged area by learning the latent patterns and structural features of the video to achieve the purpose of video repair. Figure 17 This is a schematic diagram illustrating the effect of modifying a specific area using the method embodiment ten of this application. For example... Figure 17 As shown, the first row is the original image, consisting of adjacent video frames showing the back of an elderly person. The second row consists of mask images corresponding to specific areas (damaged areas). The third row corresponds to the target images after repairing the specific areas. It can be seen that the original image of the back shows a mosaic effect due to damage, while repairing the mask image areas yields a clear target image.

[0121] In practical implementation, the solutions in this application can also allocate different computing power according to the specific areas in the original image that need to be repaired. For example... Figure 17 As shown, different computing power can be flexibly allocated according to frames at different times, so as to maximize the generation efficiency while meeting image quality requirements.

[0122] In view of the technical solutions disclosed in the above method embodiments, this application also proposes an image editing device. Figure 18 This is one embodiment of the apparatus for image editing in this application. Figure 18 As shown, the device includes: a diffusion model module 1801, a first model module 1802, and a second model module 1803. Wherein:

[0123] The diffusion model module 1801 is used to perform diffusion processing on the original image, and to perform noise reduction processing on the image after diffusion processing according to the number of noise reduction iterations to obtain the edited target image.

[0124] The first model module 1802 is used to determine the total number of noise reduction iterations based on the mask image and the occluded image.

[0125] The second model module 1803 is used to sample the network modules of the noise prediction network in the diffusion model during each noise reduction process, and the selected network modules are used to calculate the noise reduction process.

[0126] In other words, the diffusion model module 1801 performs diffusion processing on the original image and performs noise reduction processing on the diffused image according to the number of noise reduction iterations to obtain the edited target image. In the diffusion processing stage of the diffusion model module 1801, the first model module 1802 determines the total number of noise reduction iterations based on the mask image and the occlusion image. In each noise reduction process, the second model module 1803 samples the network module of the noise prediction network in the diffusion model, and the network module selected by the sampling calculates the noise reduction processing.

[0127] By applying the scheme of this application, since a first model module 1802 and a second model module 1803 are added to the diffusion model module 1801, the first model module 1802 determines the total number of denoising iterations, and the second model module 1803 samples the network modules of the noise prediction network. Therefore, when editing images, the number of iterations and the number of network modules involved in the denoising calculation can be reduced for different situations. In other words, the embodiments of this application can adaptively select appropriate iterations and appropriate network modules for different images, thereby greatly avoiding lengthy iteration processes and complex network structures while ensuring image quality, and effectively improving the generation efficiency of image editing.

[0128] This application also provides a computer-readable medium storing instructions that, when executed by a processor, can perform the steps of the image editing method described above. In practical applications, the computer-readable medium may be included in the device / apparatus / system described in the above embodiments, or it may exist independently without being assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, can implement the image editing methods described in the above embodiments. According to the embodiments disclosed in this application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0129] This application also provides a computer program product including computer instructions that, when executed by a processor, perform the methods described in any of the above embodiments.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0132] This document uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of these embodiments are merely illustrative of the method and core concepts of the present invention and are not intended to limit this application. Those skilled in the art can make changes to the specific implementation methods and application scope based on the ideas, spirit, and principles of the present invention. Any modifications, equivalent substitutions, or improvements made should be included within the scope of protection of this application.

Claims

1. An image editing method, characterized in that, The method includes: The original image is diffused using a diffusion model; The mask image and the occluded image are used as input to the first model, and the total number of noise reduction iterations is determined using the pre-trained first model. The diffusion model performs noise reduction on the diffused image according to the number of noise reduction iterations. In each noise reduction process, the network module of the noise prediction network in the diffusion model is sampled using a pre-trained second model. The network module selected by the sampling calculates the noise reduction and obtains the edited target image.

2. The method according to claim 1, characterized in that, The input to the first model further includes a guiding scale, which is a hyperparameter.

3. The method according to claim 1, characterized in that, The diffusion process includes: The original image is encoded to obtain the latent features of the original image, and the original image is mapped from the pixel space to the latent space; The occluded image is encoded to obtain the occluded image latent features, and the occluded image is mapped from the pixel space to the latent space; The original image latent features are subjected to noise processing to obtain noisy image latent features that conform to a standard normal distribution; The initial noise image latent features are obtained by stitching together the noisy image latent features, the occluded image latent features, and the mask image.

4. The method according to claim 3, characterized in that, The noise reduction process includes: The initial noisy image latent features are used as input to the current noisy image latent features of the noise prediction network; The noise prediction network is used to perform noise prediction processing to determine the noise data features of the current iteration. Then, the latent features of the current noise image are denoised based on the noise data features of the current iteration to obtain the latent features of the noise image after denoising. The noise image latent feature after this denoising is used as the new current noise image latent feature, and the process is returned to the step of using the noise prediction network for noise prediction processing until the total number of denoising iterations is completed, so as to obtain the final noise image latent feature. The latent features of the final noisy image are decoded and mapped back from the latent space to the pixel space to obtain the edited target image.

5. The method according to claim 2, characterized in that, The step of determining the total number of noise reduction iterations using a pre-trained first model includes: The mask image, the occlusion image, and the guiding scale are used as inputs to the first model; The computation is performed using the network layers of the first model, which include a first convolutional layer, a first pooling layer, a first linear layer, and a first activation function. The total number of noise reduction iterations is determined based on the output of the first model.

6. The method according to claim 1, characterized in that, The step of sampling the network modules of the noise prediction network in the diffusion model using a pre-trained second model includes: The noise prediction network is pre-divided into multiple network modules according to the resolution variation; The current noise image of the noise prediction network, the total number of noise reduction iterations, the current number of noise reduction iterations, and the mask image are used as inputs to the second model; The network layers of the second model are used for calculation to obtain the calculation results of the network layers of the second model. The network layers of the second model include a second convolutional layer, a second pooling layer, a second linear layer, and a second activation function. The probability of each network module in the noise prediction network being selected is determined based on the calculation results of the network layer of the second model. The sampling result is determined based on the probability of the network module being selected.

7. The method according to claim 6, characterized in that, The step of using the network layer of the second model to perform calculations and obtain the calculation results of the network layer of the second model includes: The current noisy image is calculated using the second convolutional layer to obtain a temporary convolution result; The temporary convolution result, the total number of noise reduction iterations, the current number of noise reduction iterations, and the mask image are fused to obtain the fusion result. The fusion result is calculated using the second convolutional layer, the second pooling layer, the second linear layer, and the second activation function to obtain the calculation result of the network layer of the second model.

8. The method according to claim 1, characterized in that, The diffusion model was trained in advance using an end-to-end approach. The total loss function for end-to-end training is expressed as L. Total =λ1L SD +λ2L FLOPs The total loss function L Total Including the first loss L SD Second loss L FLOPs The first loss L SD The second loss L is the loss of the diffusion model itself. FLOPs The loss is the sum of the losses introduced by the first model and the second model; λ1 and λ2 are hyperparameters used for the weights; the second loss is... Wherein, the F dyanmic F represents the computational cost of the first model and the second model participating in the calculation. static This indicates the computational cost involved in the calculation by the diffusion model itself.

9. An image editing device, characterized in that, The device includes: The diffusion model module is used to perform diffusion processing on the original image, and then to perform noise reduction processing on the diffused image according to the number of noise reduction iterations to obtain the edited target image. The first model module is used to determine the total number of noise reduction iterations based on the mask image and the occluded image; The second model module is used to sample the network modules of the noise prediction network in the diffusion model during each noise reduction process, and to calculate the noise reduction process by the network modules selected by the sampling.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instructions are executed by the processor, they can implement the steps of the image editing method according to any one of claims 1 to 8.

11. A computer program product comprising computer instructions that, when executed by a processor, implement the image editing method as described in any one of claims 1 to 8.

Images