Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: receiving selection operation of a user on regions in a to-be-processed image and a reference image, and obtaining a to-be-processed region in the to-be-processed image and a reference region in the reference image; based on the to-be-processed region, the reference region and the reference image, determining features used for acquiring a processed region, and based on the features, performing denoising operation on a noise map to obtain a processed region image, the noise map being obtained through the to-be-processed image and a random noise map; and determining a processed image based on the processed regional image and a non-to-be-processed region in the to-be-processed image. Through the image processing method and device, the processed region image can be generated based on the region selected by the user and the features in the unselected region in the reference image, and the image processing effect is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing and artificial intelligence, and particularly relates to an image processing method and device, an electronic device and a storage medium. BACKGROUND

[0002] Controllable region image editing based on images has become a prominent and popular research topic in the field of Artificial Intelligence Generated Content (AIGC), especially the conditional human image generation technology.

[0003] In the related art, image processing such as segmentation, generation or editing of input images to obtain processed images mainly relies on a Generative Adversarial Networks (GAN) model. However, the processed images obtained in the related art have problems such as unnatural local regions, image distortion or blurred image quality. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides an image processing method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including: receiving a selection operation of a user on a region in a to-be-processed image and a reference image, to obtain a to-be-processed region in the to-be-processed image and a reference region in the reference image. Based on the to-be-processed region, the reference region and the reference image, a feature for obtaining a processed region is determined, and a noise map is denoised based on the feature, to obtain the processed region image, the noise map being obtained by the to-be-processed image and a random noise map. Based on the processed region image and a non-to-be-processed region in the to-be-processed image, a processed image is determined.

[0006] In an implementation, the determining, based on the to-be-processed region, the reference region and the reference image, of the feature for obtaining the processed region includes: obtaining a mixed feature corresponding to the to-be-processed image and the reference image, and obtaining an encoded feature corresponding to the reference image; performing a fusion operation on the mixed feature and the encoded feature, and performing a compression operation on the fused feature to obtain m compressed data, the m compressed data having different compression degrees, m being a positive integer; and obtaining n features corresponding to the m compressed data, the n features being taken as the feature of the to-be-processed region, the reference region and the reference image, n being a positive integer greater than or equal to m.

[0007] In an embodiment, the denoising operation on the noise map based on the features obtains the processed region image, including: performing denoising operation on the noise map based on the features of the to-be-processed region and the reference region as a guide using a diffusion model, the diffusion model including an n-layer network structure.

[0008] In an embodiment, the noise map is obtained by fusing the to-be-processed image and i random noise maps, i being a positive integer greater than 1; the denoising operation on the noise map based on the features of the to-be-processed region and the reference region as a guide using a diffusion model obtains the processed region image, including: obtaining an image after j-th denoising of the noise map based on the diffusion model using a first denoising mode, j being a positive integer less than i, the first denoising mode being a conventional denoising mode of the diffusion model; based on obtaining the image after j-th denoising of the noise map using the first denoising mode, obtaining an image equivalent to an image after (j+k)-th denoising using the first denoising mode using a second denoising mode, k being a positive integer, j+k being less than or equal to i, the second denoising mode being an accelerated denoising mode of the diffusion model; predicting the image after (j+k)-th denoising of the noise map using the first denoising mode using a discrimination model, and determining the similarity between the predicted image and the image equivalent to the image after (j+k)-th denoising using the first denoising mode obtained using the second denoising mode; in response to the similarity being greater than or equal to a preset similarity threshold, obtaining an image after (j+k+1)-th denoising of the noise map using the first denoising mode; in response to the similarity being less than the preset similarity threshold, obtaining an image after (j+1)-th denoising of the noise map using the first denoising mode; repeating the above operations until an image after i-th denoising using the first denoising mode is obtained, and taking the image after i-th denoising as the processed region image.

[0009] In an embodiment, the processed image is determined based on the processed region image and the non-to-be-processed region in the to-be-processed image, including: performing smoothing processing on the processed region image based on the processed region image, the to-be-processed image, and the non-to-be-processed region in the to-be-processed image; and performing splicing operation on the smoothed image and the non-to-be-processed region in the to-be-processed image to obtain the processed image.

[0010] In an embodiment, in response to the reference image containing an object, the encoding features corresponding to the reference image include structural information of the object.

[0011] In an embodiment, specific regions in the to-be-processed image and the reference image cannot be selected.

[0012] According to a second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, comprising: an acquisition unit configured to receive a selection operation of a user on a region in a to-be-processed image and a reference image, and determine a to-be-processed region in the to-be-processed image and a reference region in the reference image; and a processing unit configured to determine a feature for obtaining a processed region based on the to-be-processed region, the reference region and the reference image, and perform a denoising operation on a noise map based on the feature, to obtain the processed region image, wherein the noise map is obtained by the to-be-processed image and a random noise map, and the processed image is determined based on the processed region image and a non-to-be-processed region in the to-be-processed image.

[0013] In an implementation, the acquisition unit determines the feature for obtaining the processed region based on the to-be-processed region, the reference region and the reference image in the following manner: acquires a hybrid feature corresponding to the to-be-processed image and the reference image, and acquires an encoding feature corresponding to the reference image; performs a fusion operation on the hybrid feature and the encoding feature, and performs a compression operation on the fused feature, to obtain m compressed data, wherein the m compressed data have different compression degrees, and m is a positive integer; acquires n features corresponding to the m compressed data, and takes the n features as the feature of the to-be-processed region, the reference region and the reference image, wherein n is a positive integer greater than or equal to m.

[0014] In an implementation, the processing unit performs the denoising operation on the noise map based on the feature in the following manner: for the noise map, uses a diffusion model to perform the denoising operation based on the features of the to-be-processed region and the reference region as a guide, to obtain the processed region image, wherein the diffusion model comprises an n-layer network structure.

[0015] In an embodiment, the noise map is obtained by fusing the to-be-processed image and i random noise maps, where i is a positive integer greater than 1; the processing unit uses a diffusion model to perform a denoising operation on the noise map based on features of the to-be-processed region and the reference region as a guide to obtain the processed region image in the following manner: based on the diffusion model, a first denoising manner is used to obtain an image after j-th denoising of the noise map, where j is a positive integer less than i, and the first denoising manner is a normal denoising manner of the diffusion model; based on the image after j-th denoising of the noise map obtained by using the first denoising manner, a second denoising manner is used to obtain an image equivalent to an image after (j+k)-th denoising obtained by using the first denoising manner, where k is a positive integer, j+k is less than or equal to i, and the second denoising manner is an accelerated denoising manner of the diffusion model; a discriminative model is used to predict an image after (j+k)-th denoising of the noise map obtained by using the first denoising manner, and a similarity between the predicted image and the image equivalent to the image after (j+k)-th denoising obtained by using the second denoising manner is determined; in response to the similarity being greater than or equal to a preset similarity threshold, an image after (j+k+1)-th denoising of the noise map is obtained by using the first denoising manner; in response to the similarity being less than the preset similarity threshold, an image after (j+1)-th denoising of the noise map is obtained by using the first denoising manner; the above operations are repeated until an image after i-th denoising obtained by using the first denoising manner is obtained, and the image after i-th denoising is taken as the processed region image.

[0016] In an embodiment, the processing unit determines a processed image based on the processed region image and the non-to-be-processed region in the to-be-processed image in the following manner: the processed region image is smoothed based on the processed region image, the to-be-processed image, and the non-to-be-processed region in the to-be-processed image; and the smoothed image and the non-to-be-processed region in the to-be-processed image are spliced to obtain the processed image.

[0017] In an embodiment, in response to the reference image containing an object, the encoding features corresponding to the reference image include structural information of the object.

[0018] In an embodiment, specific regions in the to-be-processed image and the reference image cannot be selected by a selection operation.

[0019] According to a third aspect of the embodiments of the present disclosure, an image processing apparatus is provided, including:

[0020] a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the image processing method in the first aspect or any one of the embodiments of the first aspect.

[0021] According to a fourth aspect of the embodiments of the present disclosure, a storage medium is provided, and the storage medium stores instructions. When the instructions in the storage medium are executed by a processor of a terminal, the terminal can perform the method in the first aspect or any one of the implementation manners of the first aspect.

[0022] The technical solution provided by the embodiments of the present disclosure can have the following beneficial effects: the features in the reference region in the reference image and the to-be-processed region and the reference region in the to-be-processed image selected by the user are extracted, and the processed image in which the to-be-processed region contains the features in the reference region and the non-to-be-processed region is unchanged is obtained based on the extracted features, thereby enhancing the sense of reality of image generation and improving the quality of the generated image.

[0023] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0025] Figure 1 An example diagram of a flow of an image processing method is shown according to an example embodiment.

[0026] Figure 2 is a flowchart of an image processing method according to an example embodiment.

[0027] Figure 3 is a flowchart of a feature acquisition method according to an example embodiment.

[0028] Figure 4 is a flowchart of a denoising method according to an example embodiment.

[0029] Figure 5 is a flowchart of a processed image acquisition method according to an example embodiment.

[0030] Figure 6 is a schematic diagram of a model for performing an image processing method according to an example embodiment.

[0031] Figure 7 is a block diagram of an image processing apparatus according to an example embodiment.

[0032] Figure 8 is a block diagram of an apparatus for image processing according to an example embodiment. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein with reference to the drawings. In the following description, the same or similar components are designated by the same or similar reference numerals, and repeated description of which will be omitted. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present disclosure.

[0034] In the drawings, the same or similar reference numerals are used to designate the same or similar elements throughout the drawings. The described embodiments are part of, but not all, embodiments consistent with the present disclosure. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present disclosure, and are not to be understood as limiting the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present disclosure. The embodiments of the present disclosure are described in detail below with reference to the drawings.

[0035] The image processing method provided by the embodiments of the present disclosure is applied to the fields of image processing and artificial intelligence. The image processing method provided by the embodiments of the present disclosure is mainly used for image editing based on a controllable region of an image, such as one-key dressing, image repair, and photograph editing operations. Image editing based on a controllable region of an image has become a prominent and popular research topic in the field of AIGC, especially focusing on conditional human image generation. By taking a human image and a target image as inputs, such as using a model to try a region image in the e-commerce field, a large amount of manpower and material resources are required in actual operation, but the goal of this task based on AI is to use AI to generate a photo of a person wearing a required local region seamlessly. This kind of task has requirements for both the person and the local region, first, the face, skin, and posture and identity should be the same as the person; second, the target local region is naturally deformed and seamlessly worn on the body without losing features such as patterns and textures.

[0036] In the related art, the image editing method based on a controllable region of an image mainly uses a generative adversarial network (GAN) to process images, which distorts a reference image to match a given target image, and combines the distorted image with a human image into a generator for synthesis. However, methods such as FlowNet network structure using GAN for image processing are not flexible enough and cannot handle challenging images or human poses. In addition, the results generated by the image-based GAN method usually lack a certain degree of realism and may not be able to generate more fine details. Therefore, the method of using a generative adversarial network for image processing cannot solve the problems of unnatural deformation of a local region and blurred generation quality, and this problem may bring more serious image distortion problems in actual terminal applications.

[0037] Compared with using a generative adversarial network for image processing operations, diffusion models have been widely applied in the field of image generation and have shown excellent performance in various tasks such as super-resolution, image restoration, and text-guided image generation. Compared with GAN models, diffusion models exhibit enhanced stability during training and are good at generating images with fine-grained realism, such as clearer hands and arms of a person. However, when applying diffusion models to controllable region image editing tasks, the controllability of the generated results, especially the preservation of complex textures and patterns in the target local region, remains challenging. In addition, generating high-fidelity images through diffusion models requires a large number of sampling steps, which limits their application in real-time controllable region image editing scenarios. Therefore, relying solely on diffusion models for controllable region image editing tasks, diffusion models generate high-fidelity images that require a large number of sampling steps, which limits their application in real-time controllable region image editing scenarios. Although controllability has been improved, diffusion-based methods still have the problem of redundant sampling steps.

[0038] Therefore, the embodiments of the present disclosure provide an image processing method, which obtains feature information contained in a part of a region in a to-be-processed image and a reference image, performs a denoising operation on a noise image based on the feature information, and obtains a processed image, thereby realizing image processing operation on a part of a region in the to-be-processed image based on a part of a region in the reference image, and obtaining a processed image containing features of the reference image and the to-be-processed image.

[0039] In an example embodiment, the image processing method provided by the embodiments of the present disclosure is as shown in Figure 1 Figure 1 An example diagram of the image processing method flow shown in the example embodiment.

[0040] In Figure 1 , the to-be-edited image (i.e., the to-be-processed image) and the reference image are input into an application containing the image processing method provided by the embodiments of the present disclosure, the application performs region segmentation and provides selectable regions for the user, and after the user performs a selection operation on the to-be-edited image and the reference image, the application performs image replacement on the selected region in the to-be-processed image based on the selected region and the feature information contained in other regions in the reference image, and obtains a target image (i.e., a processed image). For example, the user inputs a reference image containing clothing A and a to-be-processed image containing a person wearing clothing B, and through the image processing method provided by the embodiments of the present disclosure, if the user selects clothing A in the reference image and clothing B in the to-be-processed image, a person wearing clothing A in the to-be-processed image can be obtained.

[0041] ​It should be understood that the application scenarios of the above image processing method are only used for illustrative description, and the image processing method provided by the embodiments of the present disclosure can also be applied to image repair or photograph editing scenarios. For example, it can be used for repairing old photos, damaged images, or removing noise, stains, and the like in images. For the photographed photos, the unsatisfactory part or the part that wants to be modified to other content can be edited, and the local modification and editing of the photo content can be realized by uploading a reference image.

[0042] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment, as shown in Figure 2 , comprising the following steps.

[0043] In step S11, a to-be-processed image and a reference image are obtained.

[0044] In the embodiments of the present disclosure, the to-be-processed image and the reference image include part features used for generating a processed region image. The to-be-processed image includes a region that needs to be processed and a region that does not need to be processed. The reference image includes a region that provides indication information for the region that needs to be processed in the to-be-processed image. It should be understood that the region that needs to be processed in the to-be-processed image can be all regions in the to-be-processed image, at this time, the region that does not need to be processed can not be set or can be set as a blank image.

[0045] In step S12, a user's selection operation on the regions in the to-be-processed image and the reference image is received, and a to-be-processed region in the to-be-processed image and a reference region in the reference image are obtained.

[0046] In the embodiments of the present disclosure, the to-be-processed image and the reference image can be subjected to segmentation processing, and the segmentation regions corresponding to the to-be-processed image and the reference image are obtained. The region corresponding to the to-be-processed image selected by the user is the to-be-processed region, and the region corresponding to the reference image selected by the user is the reference region.

[0047] It should be understood that the above-mentioned method of determining the to-be-processed region and the reference region is only used for illustrative description, and the processing region and the reference region can also be the region content selected by the user through the framing, smearing, or the like in the to-be-processed image and the reference image. The embodiments of the present disclosure do not limit the method of determining the to-be-processed region and the reference region.

[0048] It should be understood that in order to ensure safety, the user cannot perform selection operations on specific regions in the to-be-processed image and the reference image, for example, cannot perform selection operations on face information in the to-be-processed image or the reference image.

[0049] In step S13, based on the to-be-processed region, the reference region and the reference image, a feature used to obtain the processed region is determined, and a denoising operation is performed on the noise map based on the feature to obtain the processed region image.

[0050] In the embodiments of the present disclosure, the processed region image is image content in the to-be-processed region in the to-be-processed image.

[0051] In the embodiments of the present disclosure, the obtained feature contains feature information in the to-be-processed region, the reference region and the reference image. The feature can be a two-dimensional matrix, a feature map or a multi-dimensional matrix, etc. For example, the feature can be a plurality of groups of vectors.

[0052] In the embodiments of the present disclosure, the denoising operation on the noise map can be performed using a pre-trained diffusion model. It should be understood that the input content of the diffusion model is text and image, and the text and image are used as guidance to perform multiple denoising operations on the noise map to obtain an output image matched with the text and image.

[0053] In the embodiments of the present disclosure, the feature can be used as guidance information to perform the denoising operation on the noise map to obtain a processed region image matched with the feature.

[0054] In the embodiments of the present disclosure, the noise map is obtained by performing a noise adding operation on the to-be-processed image and a random noise map, and the output content of the diffusion model can be an image with the same size as the noise map and the to-be-processed image. The processed region image is obtained by segmenting and cutting the image.

[0055] In step S14, based on the processed region image and the non-to-be-processed region in the to-be-processed image, a processed image is determined.

[0056] In the embodiments of the present disclosure, the non-to-be-processed region is a region in the to-be-processed image other than the to-be-processed region. For example, the to-be-processed image is composed of region A and region B. If region A is selected by a user and determined as the to-be-processed region, then region B is the non-to-be-processed region. The processed region image and the non-to-be-processed region in the to-be-processed image are spliced and fused to obtain a processed image in which part of the content in the to-be-processed image is replaced.

[0057] In the embodiments of the present disclosure, the user can select a region in the to-be-processed image and a region in the reference image, and the text information of the generated image does not need to be input by the user, so that the partial region in the to-be-processed image can be processed to obtain a partial region processed image with reference region characteristics. The controllability is enhanced by rebuilding the local region adjustment diffusion model and combining multiple control methods. The accuracy of the generated image is improved, and the situation that the generated image cannot meet the user's demand due to the deviation of the input text content is avoided. Moreover, the feature extraction of the local region image is improved, and more detailed local region information is provided to the model to better control the pattern generation in the controllable generated image.

[0058] In the embodiments of the present disclosure, the features can be determined by the mixed features of the to-be-processed image and the reference image and the features of the reference image.

[0059] Figure 3 FIG. 1 is a flowchart of a feature acquisition method according to an exemplary embodiment, as shown in FIG. 1, comprising the following steps. Figure 3

[0060] In step S21, the mixed features corresponding to the to-be-processed image and the reference image are acquired, and the encoding features corresponding to the reference image are acquired.

[0061] In the embodiments of the present disclosure, the mixed features corresponding to the to-be-processed region and the reference region and the encoding features corresponding to the reference image can be acquired by convolution, sequence processing model (Transformer) or conventional encoder, etc. For example, an encoder composed of one two-dimensional convolution layer (Conv2D), two Transformer models and one fully connected layer (Fully Connected Layer, FC) can be set to extract features of the region. For another example, a variational autoencoder (VAE-Encoder) can be set to encode the region to obtain the encoding features contained in the content of the reference image.

[0062] In the embodiments of the present disclosure, if the reference image contains an object, the structure information of the object is acquired. The object is an image content with a structure, such as a person, an animal or a car, etc. Different objects can correspond to different structure information. For example, the structure information corresponding to a person is bone posture information. The structure information is encoded together with other features, so that the encoding features corresponding to the reference image include the structure information of the object.

[0063] In step S22, the mixed features and the encoding features are fused, and the fused features are compressed to obtain m compressed data.

[0064] ​In the embodiments of the present disclosure, the fusion operation on the mixed feature and the encoded feature can be a splicing operation on the mixed feature and the encoded feature, for example, splicing the mixed feature of 512*512*1 and the encoded feature of 512*512*1 to obtain a feature of 512*1024*1. The fusion operation on the mixed feature and the encoded feature can also be a superposition operation on the mixed feature and the encoded feature in the channel dimension, for example, superimposing the mixed feature of 512*512*1 and the encoded feature of 512*512*1 to obtain a feature of 512*512*2. It should be understood that the above fusion operation on the mixed feature and the encoded feature is only used for exemplary description, and the operation mode of the fusion of the mixed feature and the encoded feature can also be determined based on actual operation requirements, and the fusion operation on the mixed feature and the encoded feature is not limited in the embodiments of the present disclosure.

[0065] In the embodiments of the present disclosure, a compression operation can be performed using a compression and excitation block (SE-block) composed of a convolutional neural network (CNN) layer, a linear layer, and a sigmoid activation function.

[0066] In the embodiments of the present disclosure, the m compressed data have different compression degrees, and m is a positive integer. The m compressed data obtained are compressed data obtained by performing m compression operations on the fused feature. For example, the first compressed data obtained is compressed data obtained by performing one compression operation on the fused feature, and the mth compressed data obtained is compressed data obtained by performing m compression operations on the fused feature.

[0067] In step S23, n features corresponding to the m compressed data are obtained, and the n features are taken as features of the to-be-processed region and the reference region.

[0068] In the embodiments of the present disclosure, n is a positive integer greater than or equal to m. The n features corresponding to the m compressed data can be obtained by a preset encoder. For example, the compressed data suitable for controlling the dimensions of different layers in a control network model (ControlNet) are input into the ControlNet model, and n vectors of different dimensions are output.

[0069] In the embodiments of the present disclosure, the n vectors obtained can be used as guide information for a denoising operation, and a diffusion model is used to perform a denoising operation on the noise map.

[0070] In the embodiments of the present disclosure, for the noise map, a pre-trained diffusion model can be used to perform a denoising operation based on the features of the to-be-processed region and the reference region as guides to obtain a processed region image.

[0071] In the embodiments of the present disclosure, if the diffusion model is used for denoising operation, n vectors are obtained using the ControlNet model, and the controlNet model can be the same as the encoder in the diffusion model, that is, the number of convolution layers, the structure and the parameters of the encoder in the diffusion model can be set to be the same as those in the controlNet model, so as to reduce the training cost. In addition, the ControlNet architecture is used to provide more content representations irrelevant to local regions as control conditions.

[0072] In the embodiments of the present disclosure, the truncated acceleration strategy including the GAN model can be used to reduce the number of denoising operations on the noise map using the diffusion model, and to improve the problem of precision reduction caused by reducing the denoising operation.

[0073] In the embodiments of the present disclosure, the noise map is obtained by fusing the to-be-processed image and i random noise maps, where i is a positive integer greater than 1, that is, the noise map is obtained by fusing the to-be-processed image with i random noise maps.

[0074] It should be understood that if the denoising operation is not accelerated using the denoising acceleration strategy, the number of denoising operations in the diffusion model should be the same as the number of noise operations performed on the obtained noise map. However, too many denoising operations will put too much pressure on the computing power of the electronic device, and there is a problem of denoising time process.

[0075] However, in the related denoising acceleration strategy, taking the denoising diffusion probabilistic model (DDPM) as an example, for the traditional DDPM, when the number of denoising steps T is reduced, the real denoising distribution q(xt-1|xt) is not approximately Gaussian, and it is usually difficult to handle. Although the denoising diffusion implicit model introduces a non-Markov diffusion process to accelerate sampling, it still needs dozens of steps to generate high-quality images. Therefore, even if a trained model is used for inference, a large amount of computing resources is required, thereby limiting the research and application of the diffusion model. The core idea of the truncated method is to integrate the idea of GAN into the diffusion model, and to use the diffusion model as an implicit generator to generate the starting point of the reverse diffusion chain, which is much shorter than the reverse diffusion chain of DDPM. However, such an approach can result in an unknown corrupted data distribution at step T’ (T’ << T), and it is usually difficult to solve by reusing the parameters of the diffusion model.

[0076] Figure 4 is a flowchart of a denoising method according to an example embodiment, as shown in Figure 4 includes the following steps.

[0077] In step S31, an image after j-th denoising of the noise map is obtained based on the diffusion model using the first denoising manner.

[0078] In the embodiments of the present disclosure, j is a positive integer less than i, and the first denoising manner is a conventional denoising manner of the diffusion model, that is, the diffusion model normally needs to perform i times of denoising operation on the noise map to obtain the processed region image.

[0079] In step S32, based on the image after j-th denoising of the noise map obtained using the first denoising manner, an image after (j+k)-th denoising obtained using the first denoising manner is obtained using the second denoising manner.

[0080] In the embodiments of the present disclosure, k is a positive integer, and j+k is less than or equal to i. After obtaining the image after j-th denoising of the noise map using the first denoising manner, the second denoising manner capable of accelerating denoising can be used to perform denoising operation on the image after j-th denoising to obtain an image after (j+k)-th denoising, which is equivalent to the image after (j+k)-th denoising obtained using the first denoising manner. For example, using a jump acceleration strategy, the result equivalent to multiple conventional denoising operations can be obtained through one denoising operation.

[0081] In the embodiments of the present disclosure, the use of the jump acceleration strategy may sacrifice a certain generation quality, because jumping multiple steps may introduce more errors and uncertainties.

[0082] In step S33, the image after (j+k)-th denoising of the noise map using the first denoising manner is predicted using the discrimination model, and the similarity between the predicted image and the image after (j+k)-th denoising obtained using the second denoising manner is determined.

[0083] In the embodiments of the present disclosure, to avoid that the denoised image obtained using the second denoising manner generates too large error and thus affects the image quality, the image after (j+k)-th denoising obtained using the first denoising manner can be predicted using the pre-trained discrimination model, and the similarity between the predicted image and the image after (j+k)-th denoising obtained using the second denoising manner is compared to determine whether the denoised image obtained using the second denoising manner generates too large error.

[0084] In the embodiments of the present disclosure, the discrimination model can be a generative adversarial model (GAN), or other models having image generation capability and similarity determination.

[0085] In step S34a, in response to the similarity being greater than or equal to a preset similarity threshold, an image after (j+k+1)-th denoising of the noise map is obtained using the first denoising manner.

[0086] In the embodiments of the present disclosure, if the similarity degree is greater than or equal to the preset similarity threshold, it is indicated that the image obtained by using the second denoising manner is equivalent to the image obtained by using the first denoising manner for j+k times of denoising, and can be used to replace the image obtained by using the first denoising manner for j+k times of denoising, that is, it is not necessary to obtain the image for j+1 to j+k times of denoising by using the first denoising manner, and the image for j+k+1 times of denoising of the noise image is obtained by using the first denoising manner.

[0087] In step S34b, in response to the similarity degree being less than the preset similarity threshold, the image for j+1 times of denoising of the noise image is obtained by using the first denoising manner.

[0088] In the embodiments of the present disclosure, if the similarity degree is less than the preset similarity threshold, it is indicated that the image obtained by using the second denoising manner is not equivalent to the image obtained by using the first denoising manner for j+k times of denoising, and cannot be used to replace the image obtained by using the first denoising manner for j+k times of denoising, that is, it is still necessary to obtain the image for j+1 times of denoising by using the first denoising manner.

[0089] In step S35, the above operation is repeated until the image for i times of denoising is obtained, and the image for i times of denoising is taken as the processed region image.

[0090] In an example embodiment, assuming that the noise image is obtained by adding noise to the image to be processed and the random noise image 40 times, the diffusion model should perform 40 denoising operations using the first denoising method. Assuming that the effect of the second denoising method is equivalent to performing the first denoising method 5 times, the prediction model is a GAN model, and the similarity threshold is 90%. For ease of description, the number of denoising operations using the first denoising method in the subsequent denoising process will be referred to as the equivalent number of times. In response to the first denoising operation using the first denoising method, an image with an equivalent number of times of 1 is obtained, and the second denoising method is used to denoise to obtain an image with an equivalent number of times of 6. The GAN model is used to predict based on the image with an equivalent number of times of 1 to obtain a predicted image with an equivalent number of times of 6. The predicted image with an equivalent number of times of 6 is compared with the image with an equivalent number of times of 6 obtained using the second denoising method. If the similarity is detected to be 91%, the first denoising method is used to obtain an image with an equivalent number of times of 7 based on the image with an equivalent number of times of 6, and the second denoising method is used to obtain an image with an equivalent number of times of 12 based on the image with an equivalent number of times of 7. The GAN model is used to obtain a predicted image with an equivalent number of times of 12 based on the image with an equivalent number of times of 7, and the similarity is compared with the image with an equivalent number of times of 12 obtained using the second denoising method. If the similarity is detected to be 81%, the first denoising method is used to obtain an image with an equivalent number of times of 8 based on the image with an equivalent number of times of 7, and the second denoising method is used to obtain an image with an equivalent number of times of 13. This process is repeated until an image with an equivalent number of times of 40 is obtained, and the image is used as the processed regional image.

[0091] In an example embodiment, the local region controllable replacement is generated by a local region diffusion model and a pre-trained GAN-based controllable region image editing model. Once the GAN model is trained, it is quantized to an int8 model and compiled into an end-side lib library. In use, the controllable generated image x to be edited is initialized directly using the library, and noise is then added through the following equation:

[0092]

[0093] In the local region controllable replacement generation, xT' is taken as the starting point of the reverse diffusion chain, and then the noise image xT' is iteratively denoised through the local region diffusion model. Compared with DDPM, t no longer follows a uniform distribution on {1, 2,..., T}, but obeys a uniform distribution on {1, 2,..., T'}. By adjusting the size of T' in the local region controllable replacement generation, the contribution ratio of the pre-trained GAN and the local region diffusion model to the final generated image can be controlled. Generally speaking, a larger T' will result in a greater influence of the local region diffusion model on the final image, while a smaller T' will make the final image more inclined to the result generated by the pre-trained GAN-based model. In this way, the GAN model replaces the noise generation method in the original DDPM, and the noise denoising process is completed before virtual noise generation, which is equivalent to a "cut-off" operation on the noise, thereby improving the production speed.

[0094] In the embodiments of the present disclosure, the inference speed is improved by the above denoising acceleration strategy, and the performance degradation when the number of sampling steps is small is avoided, and the artifacts existing in the conventional technology are corrected and the generation ability of the local region diffusion model is preserved, thereby improving the user experience.

[0095] In the embodiments of the present disclosure, the processed region image can be smoothed to ensure that the processed region image can be smoothly embedded in the non-processing region of the to-be-processed image to obtain a processed image, and an implicit distribution is used as an initial sample for the reverse denoising process instead of using Gaussian noise as the starting point of the reverse denoising, thereby significantly reducing the sampling steps.

[0096] Figure 5 is a flowchart of a processed image acquisition method according to an exemplary embodiment, as shown in Figure 5 , comprising the following steps.

[0097] In step S41, the processed region image is smoothed based on the processed region image, the to-be-processed image and the non-processing region in the to-be-processed image.

[0098] In the controllable region image editing application, it is generally desired to reposition the region image within the specified Mask mask region while keeping the region outside the Mask unchanged.

[0099] In the related art, a mask is used to connect the input image with the generated image. However, the related art can cause obvious discontinuity at the junction of the two images.

[0100] In the embodiments of the present disclosure, Poisson blending calculation is adopted to seamlessly integrate the input image and the generated image. Assuming that a controllable region image editing image directly generated, an original person image h and a non-local region Ω are given. A mixed image I p The following equation should be satisfied:

[0101]

[0102] where X p is a set of 8 connected neighbors in 8 directions around pixel p, |X p | represents the number of pixels in set X p , and table represents the boundary around Ω, is an adaptive factor calculated according to the variance of the necklace pixels. The difference between pixel p and its adjacent pixels is represented as V pp′ = h p -h p′ , where h p and h p′ represent the pixel values of pixel p and pixel p', respectively.

[0103] In step S42, the smoothed image and the non-to-be-processed region in the to-be-processed image are spliced to obtain a processed image.

[0104] In the embodiments of the present disclosure, by performing smoothing processing on the processed region image, not only the region outside the local region remains unchanged, but also the splicing trace caused by image splicing is solved. Moreover, the image region outside the local region remains unchanged, and the original image and the generated controllable generated image are seamlessly integrated.

[0105] In an exemplary embodiment, the image processing method provided by the embodiments of the present disclosure is as shown in Figure 6 . Figure 6 is a schematic diagram of a model for performing the image processing method according to an exemplary embodiment.

[0106] In Figure 6In particular, the model for performing the image processing method can be referred to as a local region diffusion model. The ControlNet architecture is adopted, while introducing additional multiple reference control conditions, while retaining the generative ability of the diffusion model (SD1.5, also referred to as the SD model), which enhances the feature extractor to provide more detailed information about the local region, thereby improving the control of the generation of the local region. For regions outside the local region, the local region diffusion model uses Poisson mixing to ensure that the original personnel information remains unchanged. For the combination with the control model, we lock all the parameters of the diffusion model, and copy the parameters of the codec of the SD model and the Cross-Attention block in the Unet model to the ControlNet as initialization, and update the parameters of the ControlNet for gradient during the training process. This method can accelerate the training and save GPU memory, thereby reducing the demand for computing resources of the diffusion model.

[0107] In Figure 6 In particular, the model for performing the image processing method can be referred to as a local region diffusion model. The ControlNet architecture is adopted, while introducing additional multiple reference control conditions, while retaining the generative ability of the diffusion model (SD1.5, also referred to as the SD model), which enhances the feature extractor to provide more detailed information about the local region, thereby improving the control of the generation of the local region. For regions outside the local region, the local region diffusion model uses Poisson mixing to ensure that the original personnel information remains unchanged. For the combination with the control model, we lock all the parameters of the diffusion model, and copy the parameters of the codec of the SD model and the Cross-Attention block in the Unet model to the ControlNet as initialization, and update the parameters of the ControlNet for gradient during the training process. This method can accelerate the training and save GPU memory, thereby reducing the demand for computing resources of the diffusion model.

[0108] In Figure 6 In particular, the model for performing the image processing method can be referred to as a local region diffusion model. The ControlNet architecture is adopted, while introducing additional multiple reference control conditions, while retaining the generative ability of the diffusion model (SD1.5, also referred to as the SD model), which enhances the feature extractor to provide more detailed information about the local region, thereby improving the control of the generation of the local region. For regions outside the local region, the local region diffusion model uses Poisson mixing to ensure that the original personnel information remains unchanged. For the combination with the control model, we lock all the parameters of the diffusion model, and copy the parameters of the codec of the SD model and the Cross-Attention block in the Unet model to the ControlNet as initialization, and update the parameters of the ControlNet for gradient during the training process. This method can accelerate the training and save GPU memory, thereby reducing the demand for computing resources of the diffusion model.

[0109] In Figure 6In the middle, the extraction of local features is mainly divided into two branches: the combination of the original image (equivalent to the image to be processed) and the reference image, and the independent modeling of the reference image. The former can establish the relevance between the two images, improve the generalization of the edge fusion of the local image, and the latter models the reference image content independently, and establishes the pixel-level features of the reference image region information. For the two groups of images input by the user (User Input), first, the segmentation is performed through the Segment Anything Model (SAM) model, and the segmentation result is passed back to the user end to prompt the user to select the area to be modified and the area to be replaced (supporting box selection, point selection, smearing, etc.), and the user-selected area is returned to the processing flow. For the combination branch of the original image and the reference image, we reserve three groups of features (segmented original image region, segmentation image, Mask mask image) of the segmentation image and the original image for reference to introduce the reference image region content. The splicing features are encoded through a simple feature fusion module (the feature fusion module includes Conv2D+Transformer*2+FC), and the full connection layer (FC) encodes the encoded region feature ψ(g) into the space required by ControlNet. For the independent modeling branch of the reference image, after segmentation using SAM, the OpenPose model is also used for skeleton pose extraction, which can ensure the stability of the human region position, and the multiple features are encoded through VAE-Encoder, and the encoded features are superimposed with the mixed features (channel dimension splicing). In order to reduce the redundancy of the features (there is redundant modeling part of the original image and the reference image for the features of the two branches), we use the SE-block composed of traditional CNN layers, Linear layers and sigmoid activation functions for compression, and respectively compress the features to the dimensions suitable for different layers of ControlNet. After inputting into the ControlNet model, according to the same idea, the Vector of different dimensions is output, and then these features are integrated into the U-Net through the cross-attention mechanism to enhance the feature extraction ability of the local region image of the local region diffusion model, and improve its controllability at the pixel level.

[0110] In the embodiments of the present disclosure, the controllability of the diffusion model is improved by providing additional control conditions and enhancing the extraction of local region image features. In terms of acceleration, the local region controllable replacement generation uses an implicit reverse denoising process generated by a pre-trained GAN-based model. A truncated-based acceleration strategy is introduced to integrate the advantages of GAN-based models and diffusion models, and to reduce the sampling steps. Compared with previous controllable generation methods based on diffusion models, the local region controllable replacement generation not only preserves the pattern and texture details of the local region in the store, but also reduces the sampling steps without affecting the generation quality. It has superiority over GAN-based and diffusion-based methods in generating more realistic images and accurately reproducing local region patterns.

[0111] Based on the same concept, the embodiments of the present disclosure also provide an image processing apparatus.

[0112] It can be understood that the image processing apparatus provided by the embodiments of the present disclosure comprises a hardware structure and / or a software module corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present disclosure.

[0113] Figure 7 is a block diagram of an image processing apparatus 100 according to an exemplary embodiment. Referring to Figure 7 The apparatus comprises an acquisition unit 101 and a processing unit 102.

[0114] The acquisition unit 101 is configured to receive a selection operation of a user on a region in a to-be-processed image and a reference image, determine a to-be-processed region in the to-be-processed image and a reference region in the reference image.

[0115] The processing unit 102 is configured to determine a feature for obtaining a processed region based on the to-be-processed region, the reference region and the reference image, perform a denoising operation on a noise map based on the feature, to obtain a processed region image, and determine a processed image based on the processed region image and a non-to-be-processed region in the to-be-processed image, wherein the noise map is obtained by the to-be-processed image and a random noise map.

[0116] In an embodiment, the acquisition unit 101 determines the features of the processed region based on the to-be-processed region, the reference region and the reference image in the following manner: acquires mixed features corresponding to the to-be-processed image and the reference image, and acquires encoding features corresponding to the reference image; performs a fusion operation on the mixed features and the encoding features, and performs a compression operation on the fused features to obtain m compressed data, the m compressed data having different compression degrees, m being a positive integer; acquires n features corresponding to the m compressed data, and takes the n features as the features of the to-be-processed region, the reference region and the reference image, n being a positive integer greater than or equal to m.

[0117] In an embodiment, the processing unit 102 performs a denoising operation on the noise graph based on the features to obtain the processed region image in the following manner: uses a diffusion model to perform a denoising operation on the noise graph based on the features of the to-be-processed region and the reference region as a guide to obtain the processed region image, the diffusion model including n layers of network structure.

[0118] In an embodiment, the noise graph is obtained by fusing the to-be-processed image and i random noise graphs, i being a positive integer greater than 1; the processing unit 102 performs a denoising operation on the noise graph using a diffusion model based on the features of the to-be-processed region and the reference region as a guide to obtain the processed region image in the following manner: based on the diffusion model, uses a first denoising manner to obtain an image after j-th denoising of the noise graph, j being a positive integer less than i, the first denoising manner being a conventional denoising manner of the diffusion model; based on the image after j-th denoising of the noise graph obtained using the first denoising manner, uses a second denoising manner to obtain an image equivalent to an image after (j+k)-th denoising obtained using the first denoising manner, k being a positive integer, j+k being less than or equal to i, the second denoising manner being an accelerated denoising manner of the diffusion model; uses a discrimination model to predict an image after (j+k)-th denoising of the noise graph using the first denoising manner, and determines a similarity degree between the predicted image and the image equivalent to the image after (j+k)-th denoising obtained using the first denoising manner; in response to the similarity degree being greater than or equal to a preset similarity threshold, uses the first denoising manner to obtain an image after (j+k+1)-th denoising of the noise graph; in response to the similarity degree being less than the preset similarity threshold, uses the first denoising manner to obtain an image after (j+1)-th denoising of the noise graph; repeats the above operations until an image after i-th denoising equivalent to the image obtained using the first denoising manner is obtained, and takes the image after i-th denoising as the processed region image.

[0119] In an embodiment, the processing unit 102 determines the processed image based on the processed region image and the non-to-be-processed region in the to-be-processed image in the following manner: performing smoothing processing on the processed region image based on the processed region image, the to-be-processed image and the non-to-be-processed region in the to-be-processed image; and performing a splicing operation on the smoothed image and the non-to-be-processed region in the to-be-processed image to obtain the processed image.

[0120] In an embodiment, in response to the reference image containing an object, the encoding feature corresponding to the reference image includes structural information of the object.

[0121] In an embodiment, the specific region in the to-be-processed image and the reference image cannot be selected by the selection operation.

[0122] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0123] Figure 8 FIG. 20 is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. The apparatus 200 can be a mobile phone, a computer, a digital broadcast terminal, a message communicator, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like, for example.

[0124] Referring to Figure 8 , the apparatus 200 can include one or more of the following components: a processing component 202, a memory 204, a power supply component 206, a multimedia component 208, an audio component 210, an input / output (I / O) interface 212, a sensor component 214, and a communication component 216.

[0125] The processing component 202 usually controls overall operations of the apparatus 200, such as operations associated with displaying, making phone calls, data communications, camera operations, and recording operations. The processing component 202 can include one or more processors 220 to execute instructions to complete all or part of steps of the above method. Further, the processing component 202 can include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 can include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0126] The memory 204 is configured to store various types of data to support the operation of the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phonebook data, messages, pictures, videos, and the like. The memory 204 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0127] The power component 206 provides power to the various components of the device 200. The power component 206 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 200.

[0128] The multimedia component 208 includes a screen providing an output interface between the device 200 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 200 is in an operation mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0129] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) configured to receive external audio signals when the device 200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0130] The I / O interface 212 provides an interface between the processing component 202 and peripheral interface modules, which can be a keyboard, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0131] The sensor component 214 includes one or more sensors to provide status assessments for various aspects of the device 200. For example, the sensor component 214 can detect an open / closed status of the device 200, relative positioning of components, such as a display and keypad of the device 200, a change in position of the device 200 or a component of the device 200, presence or absence of user contact with the device 200, orientation or acceleration / deceleration of the device 200, and temperature changes of the device 200. The sensor component 214 can include proximity sensor(s) configured to detect presence of nearby objects without any physical contact. The sensor component 214 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 214 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0132] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0133] In an exemplary embodiment, the device 200 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic modules to perform the above-described methods.

[0134] In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 204 including instructions, is also provided, which can be executed by the processor 220 of the device 200 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0135] It can be understood that, in the present disclosure, "multiple" refers to two or more, and other quantifiers are similar. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship. The singular form "a", "said" and "the" are also intended to include the plural form, unless the context clearly indicates otherwise.

[0136] It can be further understood that the terms "first", "second" and the like are used to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a particular order or importance. In fact, the expressions "first", "second" and the like can be completely interchangeable. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present disclosure.

[0137] It can be further understood that, unless otherwise specified, "connection" includes direct connection between the two without other components, and also includes indirect connection between the two with other elements.

[0138] It can be further understood that, although the operations in the embodiments of the present disclosure are described in a specific order in the accompanying drawings, it should not be understood as requiring the specific order or serial order shown, or requiring all the shown operations to be performed to obtain the desired results. In a specific environment, multi-tasking and parallel processing can be advantageous.

[0139] Other embodiments of the present disclosure will be apparent to those skilled in the art upon consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure following the general principles of the present disclosure and including common knowledge or conventional technical means in the art not disclosed by the present disclosure.

[0140] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: receiving a user selection operation on a region in a to-be-processed image and a reference image, obtaining a to-be-processed region in the to-be-processed image and a reference region in the reference image; based on the to-be-processed region, the reference region and the reference image, determining a feature for obtaining a processed region, and performing a denoising operation on a noise map based on the feature to obtain the processed region image, the noise map being obtained by the to-be-processed image and a random noise map; based on the processed region image and a non-to-be-processed region in the to-be-processed image, determining a processed image.

2. The image processing method of claim 1, wherein, The method comprises the following steps: obtaining a mixed feature corresponding to the to-be-processed image and the reference image, and obtaining an encoded feature corresponding to the reference image; performing a fusion operation on the mixed feature and the encoded feature, and performing a compression operation on the fused feature to obtain m compressed data, the m compressed data having different compression degrees, and m being a positive integer; obtaining n features corresponding to the m compressed data, and taking the n features as the features of the to-be-processed region, the reference region and the reference image, n being a positive integer greater than or equal to m.

3. The image processing method of claim 1 or 2, characterized in that, The method comprises the following steps: for the noise map, using a diffusion model to perform a denoising operation based on the features of the to-be-processed region and the reference region as a guide to obtain a processed region image, the diffusion model comprising an n-layer network structure.

4. The image processing method of claim 3, wherein, The noise map is obtained by fusing the to-be-processed image and i random noise maps, i being a positive integer greater than 1; The method comprises the following steps: based on the diffusion model, using a first denoising method to obtain an image after j-th denoising of the noise map, j being a positive integer less than i, the first denoising method being a conventional denoising method of the diffusion model; based on the image after j-th denoising of the noise map obtained using the first denoising method, using a second denoising method to obtain an image equivalent to an image after j+k-th denoising obtained using the first denoising method, k being a positive integer, j+k being less than or equal to i, the second denoising method being an accelerated denoising method of the diffusion model; using a discriminative model to predict an image after j+k-th denoising of the noise map using the first denoising method, and determining a similarity degree between the predicted image and the image equivalent to the image after j+k-th denoising obtained using the first denoising method using the second denoising method; in response to the similarity degree being greater than or equal to a preset similarity threshold, obtaining an image after j+k+1-th denoising of the noise map using the first denoising method; in response to the similarity degree being less than the preset similarity threshold, obtaining an image after j+1-th denoising of the noise map using the first denoising method; The above operations are repeated until an image equal to the i-th denoised image obtained using the first denoising manner is obtained, and the i-th denoised image is taken as the processed region image.

5. The image processing method of claim 1 or 3, characterized by, The processed image is determined based on the processed region image and the non-to-be-processed region in the to-be-processed image, comprising: The processed region image is smoothed based on the processed region image, the to-be-processed image and the non-to-be-processed region in the to-be-processed image; The smoothed image and the non-to-be-processed region in the to-be-processed image are spliced to obtain the processed image.

6. The image processing method of claim 2, wherein, In response to the reference image containing an object, the structure information of the object is included in the encoding feature corresponding to the reference image.

7. The image processing method of claim 1, wherein, The specific region in the to-be-processed image and the reference image cannot be selected by the selection operation.

8. An image processing apparatus characterized by comprising: Comprising: The acquisition unit receives the selection operation of the user on the region in the to-be-processed image and the reference image, determines the to-be-processed region in the to-be-processed image and the reference region in the reference image; The processing unit determines the feature used to obtain the processed region based on the to-be-processed region, the reference region and the reference image, and performs denoising operation on the noise graph based on the feature to obtain the processed region image, wherein the noise graph is obtained by the to-be-processed image and a random noise graph, and the processed image is determined based on the processed region image and the non-to-be-processed region in the to-be-processed image.

9. The image processing apparatus according to claim 8, characterized by The acquisition unit determines the feature used to obtain the processed region based on the to-be-processed region, the reference region and the reference image in the following manner: Obtain the mixed feature corresponding to the to-be-processed image and the reference image, and obtain the encoding feature corresponding to the reference image; Fuse the mixed feature and the encoding feature, and compress the fused feature to obtain m compressed data, wherein the compression degrees of the m compressed data are different, and m is a positive integer; Obtain n features corresponding to the m compressed data, and take the n features as the features of the to-be-processed region, the reference region and the reference image, wherein n is a positive integer greater than or equal to m.

10. The image processing apparatus according to claim 8 or 9, characterized by, The processing unit performs denoising operation on the noise graph based on the feature in the following manner to obtain the processed region image: For the noise graph, use the diffusion model to perform denoising operation based on the features of the to-be-processed region and the reference region as a guide to obtain the processed region image, wherein the diffusion model includes n layers of network structure.

11. The image processing apparatus according to claim 10, characterized by, The noise graph is obtained by fusing the to-be-processed image and i random noise graphs, wherein i is a positive integer greater than 1; The processing unit uses the diffusion model to perform denoising operation on the noise graph based on the features of the to-be-processed region and the reference region as a guide in the following manner to obtain the processed region image: Based on the diffusion model, a j-th denoised image of the noise graph is obtained using a first denoising manner, wherein j is a positive integer less than i, and the first denoising manner is a conventional denoising manner of the diffusion model; Based on the image obtained after the j-th denoising of the noise map using the first denoising method, the image obtained after the (j+k)-th denoising is obtained using the second denoising method, where k is a positive integer, j+k is less than or equal to i, and the second denoising method is the accelerated denoising method of the diffusion model. The discriminative model is used to predict the image after the noise map is denoised for the (j+k)th time using the first denoising method, and the similarity between the predicted image and the image obtained using the second denoising method is equivalent to the image obtained using the first denoising method after the (j+k)th time denoising. In response to the similarity being greater than or equal to a preset similarity threshold, the image after the (j+k+1)th denoising step of the noise map is obtained using a first denoising method; In response to the similarity being less than a preset similarity threshold, the image after the (j+1)th denoising of the noise map is obtained using the first denoising method; Repeat the above operation until an image equivalent to the i-th denoised image obtained using the first denoising method is obtained, and use the i-th denoised image as the processed region image.

12. The image processing apparatus according to claim 8 or 10, characterized by, The processing unit determines the processed image based on the processed region image and the non-processable region in the image to be processed in the following manner: Based on the processed region image, the image to be processed, and the non-processable region in the image to be processed, the processed region image is smoothed. The smoothed image and the non-processed area in the image to be processed are stitched together to obtain the processed image.

13. The image processing apparatus according to claim 9, characterized by, In response to the fact that the reference image contains an object, the coded features corresponding to the reference image include the structural information of the object.

14. The image processing apparatus according to claim 8, characterized by, Specific regions in the image to be processed and the reference image cannot be selected for operation.

15. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method described in any one of claims 1 to 7.

16. A storage medium, characterized by The storage medium stores instructions that, when executed by the terminal's processor, enable the terminal to perform the method described in any one of claims 1 to 7.

Images