Image processing method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to an image processing method and device, a computer readable storage medium and electronic equipment, and the method comprises the steps: obtaining a to-be-processed image and a mask image, and enabling the mask image to be used for identifying a to-be-complemented region of the to-be-processed image; according to the to-be-processed image and the mask image, performing completion processing on a to-be-completed area of the to-be-processed image by using a first image generation model to obtain a first completion image, the first image generation model being formed by training a non-diffusion model; and according to the first completion image, the to-be-processed image and the mask image, performing completion processing on a to-be-completed area in the to-be-processed image by using a second image generation model to obtain a target completion image, the second image generation model being trained by a diffusion model. Therefore, the image completion result of the non-diffusion model can be utilized to guide the diffusion model to generate an image completion result which is good in the aspects of structure completion, structure recovery, texture generation, semantic consistency and the like, so that the image completion effect is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and particularly to an image processing method, apparatus, computer-readable storage medium, and electronic device. Background Art

[0002] Image completion is a fundamental problem in computer vision, and its purpose is to fill in the missing image regions with reasonable content. Many applications have been derived based on this problem, including: image editing, image relocalization, photo restoration, and object removal, etc.

[0003] However, the current image completion methods have the problem of poor image completion effect. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, computer-readable storage medium, and electronic device to improve the image completion effect.

[0005] Embodiments of this application provide an image processing method, including:

[0006] Obtain an image to be processed and a mask image, where the mask image is used to identify the region to be completed in the image to be processed;

[0007] According to the image to be processed and the mask image, use a first image generation model to perform completion processing on the region to be completed in the image to be processed to obtain a first completed image, and the first image generation model is trained by a non-diffusion model;

[0008] According to the first completed image, the image to be processed, and the mask image, use a second image generation model to perform completion processing on the region to be completed in the image to be processed to obtain a target completed image, and the second image generation model is trained by a diffusion model.

[0009] Embodiments of this application also provide an image processing apparatus, including:

[0010] An obtaining module, configured to obtain an image to be processed and a mask image, where the mask image is used to identify the region to be completed in the image to be processed;

[0011] A first completion module, configured to perform completion processing on the region to be completed in the image to be processed according to the image to be processed and the mask image by using a first image generation model to obtain a first completed image, and the first image generation model is trained by a non-diffusion model;

[0012] A second completion module, configured to perform completion processing on the region to be completed in the image to be processed according to the first completed image, the image to be processed, and the mask image by using a second image generation model to obtain a target completed image, and the second image generation model is trained by a diffusion model.

[0013] Among them, the image processing device further includes:

[0014] A first determination module, configured to determine whether the first completed image meets the requirements;

[0015] Specifically, the second completion module is configured to:

[0016] In response to the first completed image not meeting the requirements, based on the first completed image, the image to be processed, and the mask image, use the second image generation model to perform completion processing on the area to be completed of the image to be processed, and obtain the target completed image.

[0017] Among them, the second completion module specifically includes:

[0018] A feature extraction unit, configured to extract features from the first completed image to obtain completed image features;

[0019] A first completion unit, configured to, based on the completed image features, the image to be processed, and the mask image, use the second image generation model to perform completion processing on the area to be completed of the image to be processed, and obtain the target completed image.

[0020] Among them, specifically, the first completion unit is configured to:

[0021] Input the completed image features, the image to be processed, and the mask image into the second image generation model. In the second image generation model, perform denoising processing on the target noise image according to the completed image features, the image to be processed, and the mask image, and obtain the target completed image output by the second image generation model.

[0022] Among them, the first completion unit specifically includes:

[0023] A completion subunit, configured to, based on the completed image features, the image to be processed, and the mask image, use the second image generation model to perform completion processing on the area to be completed of the image to be processed, and obtain the second completed image;

[0024] A fusion subunit, configured to fuse the first completed image and the second completed image to obtain the target completed image.

[0025] Among them, the second completion module specifically includes:

[0026] A second completion unit, configured to, based on the image to be processed and the mask image, use the second image generation model to perform completion processing on the area to be completed of the image to be processed, and obtain the second completed image;

[0027] A fusion unit, configured to fuse the first completed image and the second completed image to obtain the target completed image.

[0028] Among them, specifically, the second completion unit is configured to:

[0029] Input the image to be processed and the mask image into the second image generation model. In the second image generation model, denoise the target noise image according to the image to be processed and the mask image to obtain the second completed image output by the second image generation model.

[0030] Among them, fusing the first completed image and the second completed image to obtain the target completed image specifically includes:

[0031] Input the first completed image and the second completed image into the fusion module. In the fusion module, extract the first completed image feature of the first completed image and the second completed image feature of the second completed image, and generate the target completed image according to the first completed image feature and the second completed image feature.

[0032] Among them, the image processing device further includes:

[0033] The second determination module is used to determine whether the second completed image meets the requirements;

[0034] Fusing the first completed image and the second completed image to obtain the target completed image specifically includes:

[0035] In response to the second completed image not meeting the requirements, fuse the first completed image and the second completed image to obtain the target completed image.

[0036] Among them, the first completion module is specifically used for:

[0037] Input the image to be processed and the mask image into the first image generation model. In the first image generation model, according to the image to be processed and the mask image, use a convolutional neural network, a transformer, and / or a generative adversarial network to complete the area to be completed of the image to be processed to obtain the first completed image output by the first image generation model.

[0038] Among them, the acquisition module specifically includes:

[0039] The acquisition unit is used to acquire the image to be processed;

[0040] The determination unit is used to, in response to a region selection operation on the image to be processed, determine the region to be completed selected by the region selection operation from the image to be processed, and generate a mask image according to the region to be completed.

[0041] An embodiment of this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute any one of the above image processing methods.

[0042] An embodiment of the present application also provides an electronic device, which includes a memory and a processor. A computer program is stored in the memory, and the processor executes any one of the above image processing methods by calling the computer program stored in the memory.

[0043] The image processing method, device, computer-readable storage medium, and electronic device provided by the present application obtain a to-be-processed image and a mask image. The mask image is used to identify the to-be-completed area of the to-be-processed image, and according to the to-be-processed image and the mask image, a first image generation model is used to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a first completed image. The first image generation model is trained by a non-diffusion model. Then, according to the first completed image, the to-be-processed image, and the mask image, a second image generation model is used to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a target completed image. The second image generation model is trained by a diffusion model. Thus, it is realized that first, a non-diffusion model (such as a convolutional neural network) is used to generate an image completion result with good completion structure and semantic consistency, and then this image completion result is used to guide the diffusion model to generate a higher-quality image completion result, which can give full play to the respective advantages of the non-diffusion model and the diffusion model, so as to obtain good image completion effects in terms of completion structure, structure restoration, texture generation, and semantic consistency, etc., making the completed image have no obvious distortion and good semantic consistency, and thus can significantly improve the image completion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The technical solutions and other beneficial effects of the present application will become obvious by describing the specific embodiments of the present application in detail with reference to the accompanying drawings.

[0045] Figure 1 It is a schematic diagram of the scenario of the image processing system provided by the embodiment of the present application;

[0046] Figure 2 It is a schematic flowchart of the image processing method provided by the embodiment of the present application;

[0047] Figure 3 It is an operation schematic diagram of removing the original image of the to-be-completed area in the to-be-processed image to obtain the to-be-completed image provided by the embodiment of the present application;

[0048] Figure 4 It is another operation schematic diagram of removing the original image of the to-be-completed area in the to-be-processed image to obtain the to-be-completed image provided by the embodiment of the present application;

[0049] Figure 5 It is an operation schematic diagram of fusing the first completed image and the second completed image to obtain the target completed image provided by the embodiment of the present application;

[0050] Figure 6 Another schematic diagram of the operation of fusing the first completed image and the second completed image to obtain the target completed image provided by the embodiment of the present application;

[0051] Figure 7 Another schematic flowchart of the image processing method provided by the embodiment of the present application;

[0052] Figure 8 Another schematic flowchart of the image processing method provided by the embodiment of the present application;

[0053] Figure 9 Another schematic flowchart of the image processing method provided by the embodiment of the present application;

[0054] Figure 10 Another schematic flowchart of the image processing method provided by the embodiment of the present application;

[0055] Figure 11 Schematic diagram of the structure of the image processing device provided by the embodiment of the present application;

[0056] Figure 12 Schematic diagram of the structure of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0058] In the following description, specific embodiments of the present application will be described with reference to steps and symbols executed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times as being executed by a computer. As used herein, computer execution includes operations of a computer processing unit that represents electronic signals in a structured form of data. This operation transforms the data or maintains it in a position in the computer's memory system, which can be reconfigured or otherwise changed in a manner well known to those skilled in the art. The data structure maintained by the data is the physical location of the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation. Those skilled in the art will understand that the various steps and operations described below can also be implemented in hardware.

[0059] As used herein, the terms "module" or "unit" may be regarded as software objects executing on the computing system. The different components, modules, engines, and services described herein may be regarded as implementation objects on the computing system. The devices and methods described herein are preferably implemented in software, but may also be implemented in hardware, all within the scope of protection of this application.

[0060] In the following description of this application, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0061] In the following description of this application, the terms "first", "second", and "third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first", "second", and "third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] A detailed description will be given below in conjunction with specific embodiments. It should be noted that the serial numbers of the following embodiments do not limit the preferred order of the embodiments.

[0064] Please refer to Figure 1 , Figure 1 , which is a schematic diagram of the scenario of the image processing system provided by the embodiment of this application. The image processing system may include any one of the image processing devices provided by the embodiment of this application. The image processing device may be specifically integrated in an electronic device such as a terminal or a server. The terminal may be a smart phone, a tablet computer, a laptop computer, or a personal computer (PC), etc. The server may be a specific independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0065] The electronic device can obtain an image to be processed and a mask image, where the mask image is used to identify the region to be completed in the image to be processed; according to the image to be processed and the mask image, use a first image generation model to complete the region to be completed in the image to be processed, obtaining a first completed image, and the first image generation model is trained by a non-diffusion model; according to the first completed image, the image to be processed and the mask image, use a second image generation model to complete the region to be completed in the image to be processed, obtaining a target completed image, and the second image generation model is trained by a diffusion model.

[0066] Among them, the image to be processed can be an image captured by the camera of the user's terminal, or an image stored in the terminal, or an image obtained by the terminal from other devices. The mask image can be a binary image with the same size as the image to be processed, and the region with a value of 0 (that is, black) in the mask image can be used to identify the region to be completed in the image to be processed. Specifically, the position of the region to be completed in the image to be processed can be the same as the position of the region with a value of 0 in the mask image. Moreover, completing the region to be completed in the image to be processed can include: removing the original image in the region to be completed and filling a new image in the region to be completed. The non-diffusion model (Non Diffusion Model, NonDM) can include Convolutional Neural Networks (CNN), Transformer (XFormer), and / or Generative Adversarial Network (GAN), etc. The diffusion model (DiffusionModel, DM) can be a Latent Diffusion Model, a StableDiffusion Model, or a Denoising diffusion probalistic models (DDPM), etc. In some examples, the above diffusion model can be a diffusion model without classifier guidance (such as, Stable Diffusion Model).

[0067] In a possible application scenario, such as Figure 1As shown in the figure, the above-mentioned image processing system may include a terminal 10 and a server 20 that are communicatively connected. Specifically, when a user wants to remove unwanted images (such as people or objects, etc.) from an image to be processed, the user can select a region to be completed that includes the unwanted image from the image to be processed through operation methods such as clicking, touching, or swiping, so as to trigger the terminal 10 to generate a mask image for identifying the region to be completed. After that, the terminal 10 may send the image to be processed and the mask image for identifying the region to be completed of the image to be processed to the server 20, so as to trigger the server to receive the image to be processed and the mask image, and according to the image to be processed and the mask image, use a first image generation model to perform a completion process on the region to be completed of the image to be processed to obtain a first completed image. Then, the server may use a second image generation model to perform a completion process on the region to be completed of the image to be processed according to the first completed image, the image to be processed, and the mask image to obtain a target completed image. After that, the server 20 may also feedback the target completed image to the terminal 10 for the user to view, save, etc.

[0068] As Figure 2 shown, Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application. The specific process of this image processing method may be as follows:

[0069] S101. Obtain an image to be processed and a mask image, where the mask image is used to identify the region to be completed of the image to be processed.

[0070] In this embodiment, the image to be processed may be an image captured by a camera of a user's terminal (such as a smart phone), or an image stored in the terminal, or an image obtained by the terminal from other devices. Additionally, the image to be processed may be an image of a real scene or a virtual constructed image (such as an animation diagram, etc.). The region to be completed in the above-mentioned image to be processed may be the region where unwanted elements (such as distracting scene elements like people or objects, etc.) in the image to be processed are located.

[0071] The mask image can also be referred to as a masking image, which is mainly used to block an image region to achieve the purpose of processing only the corresponding image region. The mask image in this embodiment may be a binary image with the same size as the image to be processed, and the region to be completed in the image to be processed may be the region in the image to be processed that has the same position as the black or white region of the mask image.

[0072] For example, the area with a value of 0 (i.e., black) in the mask image may correspond to the area in the to-be-processed image that does not require image completion, and the area with a value of 1 (i.e., white) may correspond to the area in the to-be-processed image that requires image completion (i.e., the area to be completed above). Another example is that the area with a value of 1 in the mask image may correspond to the area in the to-be-processed image that does not require image completion, and the area with a value of 0 may correspond to the area in the to-be-processed image that requires image completion.

[0073] In some examples, as Figure 3 shown, the area to be completed A11 of the to-be-processed image A1 may be the area where the person is located in the to-be-processed image A1, and the black area A21 in the mask image A2 can be used to identify the area to be completed A11 in the to-be-processed image A1. In other examples, as Figure 4 shown, the area to be completed B11 of the to-be-processed image B1 may be the area where the object (such as a bag) is located in the to-be-processed image B1, and the black area B21 in the mask image B2 can be used to identify the area to be completed B11 in the to-be-processed image B1.

[0074] Moreover, in specific implementation, the number of areas to be completed in the to-be-processed image may be one or more, that is, the function of the mask image includes indicating which area or areas in the to-be-processed image need to be completed.

[0075] In some specific embodiments, the above S101 may specifically include:

[0076] S1011. Obtain the to-be-processed image.

[0077] Specifically, when the user wants to perform completion processing on a certain image (i.e., the to-be-processed image), the user can send the to-be-processed image to the above image processing device through the terminal used by the user, so that the image processing device receives the to-be-processed image and performs completion processing on the to-be-processed image.

[0078] S1012. In response to the area selection operation for the to-be-processed image, determine the area to be completed selected by the area selection operation from the to-be-processed image, and generate a mask image according to the area to be completed.

[0079] Specifically, when the user wants to perform a filling process on a specified area in the above-mentioned image to be processed, an area selection operation for the above-mentioned image to be processed can be performed. For example, the specified area can be selected from the above-mentioned image to be processed through operations such as clicking, touching, and swiping, so as to trigger the above-mentioned image processing device to respond to the user's area selection operation for the above-mentioned image to be processed, use the specified area of the image to be processed selected by the area selection operation as the area to be filled, and generate a mask image based on the area to be filled, so as to subsequently perform a filling process on the area to be filled in the image to be processed based on the mask image and the image to be processed. Thus, the user can delete the image in the specified area of the image to be processed according to their own needs, and fill in the missing image in the specified area after deletion to generate a filled image corresponding to the image to be processed, which is beneficial to improving the user experience.

[0080] S102. According to the image to be processed and the mask image, use the first image generation model to perform a filling process on the area to be filled in the image to be processed to obtain a first filled image, and the first image generation model is trained by a non-diffusion model.

[0081] Among them, performing a filling process on the area to be filled in the image to be processed may include: removing the original image in the area to be filled in the image to be processed to obtain an image to be filled, and filling a new image in the area to be filled in the image to be filled to obtain a filled image corresponding to the image to be processed.

[0082] Specifically, the above-mentioned removing the original image in the area to be filled in the image to be processed to obtain an image to be filled may specifically include: modifying the color values of each pixel point in the area to be filled in the image to be processed to 0, and keeping the color values of each pixel point in the non-area to be filled in the image to be processed (that is, the area other than the area to be filled) unchanged, so as to achieve the purpose of selectively removing the original image in the area to be filled in the image to be processed, and obtain the corresponding image to be filled.

[0083] In specific implementation, in the above-mentioned embodiment where the area with a value of 0 (that is, black) in the mask image corresponds to the area in the image to be processed that needs to be filled with an image (that is, the above-mentioned area to be filled), the above-mentioned removing the original image in the area to be filled in the image to be processed to obtain an image to be filled may specifically include: multiplying the above-mentioned image to be processed and the mask image to obtain an image to be filled, where the image to be filled can be expressed as Image*Mask, Image is the image to be processed, and Mask is the mask image. In some examples, as Figure 3 shown, multiplying the image to be processed A1 and the mask image A2 can obtain the image to be filled A3, and the area to be filled A31 of the image to be filled A3 is the area in the image to be filled A3 that is in the same position as the area to be filled A11 of the image to be processed A1. In other examples, asFigure 4 As shown, multiplying the image to be processed B1 by the mask image B2 can obtain the image to be completed B3, and the area to be completed B31 of the image to be completed B3 is the area in the image to be completed B3 that has the same position as the area to be completed B11 of the image to be processed B1.

[0084] In this embodiment, the above first image generation model can perform an image completion task, which is used to complete the area to be completed of the above image to be processed to obtain the corresponding image completion result (that is, the above first completed image).

[0085] Specifically, the above first image generation model can be a trained non-diffusion model (Non Diffusion Model, NonDM), and the non-diffusion model can include convolutional neural networks (Convolutional Neural Networks, CNN), transformers (Transformer, XFormer), and / or generative adversarial networks (Generative Adversarial Network, GAN), etc.

[0086] Correspondingly, the above S102 can specifically include: inputting the above image to be processed and the mask image into the first image generation model, and in the first image generation model, using convolutional neural networks, transformers, and / or generative adversarial networks to complete the area to be completed of the above image to be processed to obtain the first completed image output by the first image generation model.

[0087] Specifically, in the above first image generation model, the input image to be processed can be multiplied by the mask image first to remove the original image in the area to be completed of the image to be processed and obtain the image to be completed. Then, according to the image in the non-area to be completed (that is, the area other than the area to be completed) of the image to be completed, use convolutional neural networks, transformers, and / or generative adversarial networks to generate the missing image (that is, the removed image) in the area to be completed of the image to be completed, and use the generated missing image to fill the area to be completed in the above image to be completed to obtain the first completed image, thereby realizing the completion processing of the area to be completed in the above image to be processed.

[0088] In some examples, the above first image generation model can be an image generation model based on convolutional neural networks.

[0089] Specifically, adversarial training has been proven crucial for generating clear completion results. Many customized completion convolutions can be designed to simulate the general features of the occluded regions (i.e., the regions to be completed). In some specific examples, the first image generation model can be an AOT (aggregated contextual transformations) - GAN generator constructed by stacking AOT blocks, which can aggregate context transformations from various receptive fields, and the AOT - GAN can be trained using a pre - designed mask prediction task to enhance its discriminator, thereby promoting the texture synthesis effect.

[0090] Moreover, the attention mechanism can also be applied to the image completion task. That is, the first image generation model can be a convolutional neural network based on the attention mechanism, so that long - range relationships can be captured through the attention mechanism, and reliable features can be aggregated from the non - masked regions (i.e., the non - completion regions) to the masked regions (i.e., the completion regions). In some specific examples, the first image generation model can be a convolutional neural network based on the context residual aggregation (CRA) mechanism, so that the high - frequency residuals of the missing content can be generated by weighted aggregation of the residuals of the context patches. In other specific examples, the first image generation model can also be a two - stage convolutional network, which first completes through a coarse - grained network and then optimizes the completion region using a fine - grained network. Among them, the fine - grained network can include a context attention layer for pixel blocks.

[0091] In other examples, the first image generation model can be a Transformer - based image generation model.

[0092] Specifically, Transformer (i.e., XFormer) has the ability to learn long - range interactions between sequential image patches, and thus performs excellently in many visual tasks. Therefore, XFormer can be used to complete some low - resolution images and guide the CNN - based GAN to generate high - quality completion results. For example, XFormer can be used for appearance prior reconstruction and CNN for texture supplementation. Another example is that a new type of bidirectional autoregressive transformer (BAT) for image inpainting can be introduced. Another example is that the advantages of convolution and XFormer can be combined to design a completion - specific XFormer block, where the attention module can be indicated by a dynamic mask to aggregate non - local information only from part of the valid tokens. Another example is that image completion can be regarded as an undirected sequence - to - sequence (seq2seq) prediction task, and Transformer can be used to directly capture the long - range dependencies in the encoder, while introducing an attention - aware layer (AAL) to better utilize the high - frequency features related to the long - range, improving the appearance consistency between the visible region and the generated region.

[0093] In some other examples, the above-mentioned first image generation model can be an image generation model based on large-scale masked region conditions.

[0094] In some specific examples, the above-mentioned first image generation model can be a co-modulated GAN (co-modulated generative adversarial network), which establishes the association between the image conditional and unconditional generation model architectures through the co-modulation of conditional and random style representations.

[0095] In some other specific examples, the above-mentioned first image generation model can be a CM-GAN (Cross-modal Generative Adversarial Network), which uses a target-aware training mechanism to generate more realistic masks during the training process to better support real target removal use cases while preventing the model from attempting to synthesize new targets within the holes.

[0096] In some other specific examples, the above-mentioned first image generation model can be trained with a large proportion of masks, introduce fast Fourier convolution (FFC) with a wide receptive field range into the architecture of the above-mentioned first image generation model, and a relatively high receptive field perception loss can be set.

[0097] In some other specific examples, the above-mentioned first image generation model can be a jointly modulated image completion model based on StyleGAN (style-based generative adversarial network), and this jointly modulated image completion model can achieve high-quality generation effects through style learning enhancement.

[0098] In some other specific examples, the above-mentioned first image generation model can be a two-stage model (for example, it can be an image generation model based on XFormer and CNN). This two-stage model can first predict the image structure of the missing region in the form of a Canny (an edge detection algorithm) edge sketch, and then complete the image based on the predicted edge sketch.

[0099] And it should be noted that as Figures 3 to 6 shown, when the first image generation model trained by the non-diffusion model in this embodiment is used to complete the to-be-completed regions A11 / B11 of the above-mentioned to-be-processed images A1 / B1 to obtain the first completed images A4 / B4, it can relatively well ensure the semantic consistency and structural continuity between the regions in the first completed images A4 / B4 that have been completed and the regions that have not been completed, and thus can at least meet the user's needs to a certain extent.

[0100] S103. According to the first completed image, the image to be processed, and the mask image, use the second image generation model to complete the area to be completed of the image to be processed, obtaining a target completed image, where the second image generation model is trained by a diffusion model.

[0101] Specifically, after using the above first image generation model to complete the area to be completed of the above image to be processed and obtaining the first completed image, since the first completed image may not meet the user's requirements, the second image generation model can be further used on the basis of this first completed image to complete the area to be completed of the above image to be processed, so as to obtain a higher-quality final image completion result (that is, the above target completed image).

[0102] In some embodiments, as Figure 7 shown, after the above S102, the above image processing method may further include:

[0103] S104. Determine whether the first completed image meets the requirements.

[0104] Specifically, after obtaining the above first completed image, the above image processing device may send the above first completed image to a target object (for example, a user terminal). Then, the target object can receive the above first completed image and display the above first completed image in an interaction interface so that the user of the target object can view the above first completed image. Moreover, after the user of the target object views the above first completed image, the user can send first feedback information regarding the above first completed image through the above interaction interface.

[0105] Among them, the first feedback information can be used to indicate whether the above first completed image meets the requirements. Specifically, when the user believes that the above first completed image can meet their needs, the user can send, through the target object, first feedback information indicating that the above first completed image meets the requirements to the above image processing device; when the user believes that the above first completed image cannot meet their needs, the user can send, through the target object, first feedback information indicating that the above first completed image does not meet the requirements to the above image processing device.

[0106] Moreover, the above image processing device can receive the first feedback information of the target object regarding the above first completed image, and in response to the first feedback information indicating that the above first completed image does not meet the requirements, determine that the above first completed image does not meet the requirements, and in response to the first feedback information indicating that the above first completed image meets the requirements, determine that the above first completed image meets the requirements.

[0107] Accordingly, the above S103 may specifically include: in response to the first completed image not meeting the requirements, based on the first completed image, the image to be processed, and the mask image, using a second image generation model to perform completion processing on the area to be completed of the image to be processed, so as to obtain a target completed image. Thus, it is possible to further use the second image generation model to perform completion processing on the area to be completed of the above-mentioned image to be processed based on the first completed image, in order to obtain a higher-quality final image completion result (i.e., the above-mentioned target completed image), and solve the problem that the image completion result of the above-mentioned first image generation model (i.e., the above-mentioned first completed image) cannot meet the user's needs.

[0108] Moreover, when it is determined that the above-mentioned first completed image meets the requirements, the above-mentioned image processing device may not perform further processing, that is, the above-mentioned first completed image may be used as the final completion result, without further using the second image generation model to perform completion processing on the area to be completed of the image to be processed based on the first completed image, the image to be processed, and the mask image to obtain a target completed image. Therefore, it is possible to improve the image completion efficiency while ensuring that the image completion result meets the user's needs.

[0109] In this embodiment, the above-mentioned second image generation model may be a trained diffusion model (Diffusion Model, DM), and the diffusion model may be a latent diffusion model, a stable diffusion model, or a denoising diffusion probabilistic model (DDPM), etc. In some examples, the above-mentioned diffusion model may be a classifier-free guidance diffusion model (such as, Stable Diffusion Model).

[0110] It should be noted that with the emergence of the powerful generation method of diffusion models, the field of image generation has developed rapidly. Moreover, compared with the first image generation model trained by a non-diffusion model, when the second image generation model trained by a diffusion model performs an image completion task, due to the multi-step generation ability of the diffusion model, the subjective completion effect of the second image generation model is better than that of the first image generation model.

[0111] However, training a diffusion model specific to the image completion task requires a considerable amount of computing power overhead, and as many diffusion steps as possible are needed to obtain good completion results through the reverse denoising process. At the same time, most existing diffusion models rely on good text prompt information to achieve good completion performance. Therefore, compared with the first image generation model trained by a non-diffusion model, when the second image generation model trained by a diffusion model performs the image completion task, although it can obtain a better subjective completion effect, the time taken for the second image generation model to complete the image completion task will be much longer than that of the first image generation model to complete the image completion task, and the second image generation model has great uncertainty in the completion structure. The completion effect of the first image generation model in terms of completion structure and semantic consistency is better than that of the second image generation model.

[0112] Moreover, it can be understood that in this embodiment, by first using the first image generation model to perform completion processing on the area to be completed of the image to be processed to obtain the first completed image, and then using the second image generation model to perform completion processing on the area to be completed of the image to be processed based on the first completed image to obtain the target completed image, the respective advantages of the two types of models (i.e., non-diffusion models and diffusion models) can be fully utilized to obtain good completion effects in terms of completion structure, structure restoration, texture generation, and semantic consistency.

[0113] In addition, after obtaining the first completed image by using the first image generation model to perform completion processing on the area to be completed of the image to be processed, first determine whether the above first completed image meets the requirements, and when the above first completed image does not meet the requirements, further use the second image generation model to perform completion processing on the area to be completed of the image to be processed based on the above first completed image to obtain the target completed image. This is also beneficial to achieving a better balance between the image completion effect and the image completion efficiency, so as to shorten the time taken for the image completion task and improve the image completion efficiency while ensuring that the completion result can meet the user's needs, which is conducive to the user's convenient, controllable, high-quality, and intelligent editing operations on the image, making the album use more intelligent and user-friendly.

[0114] In some specific embodiments, as Figure 8 shown, the above S103, or using the second image generation model to perform completion processing on the area to be completed of the image to be processed according to the first completed image, the image to be processed, and the mask image to obtain the target completed image, may specifically include:

[0115] S1031. According to the image to be processed and the mask image, use the second image generation model to perform completion processing on the area to be completed of the image to be processed to obtain the second completed image.

[0116] In some examples, the above S1031 may specifically include: inputting the image to be processed and the mask image into a second image generation model, where in the second image generation model, denoising the target noise image according to the image to be processed and the mask image to obtain a second completed image output by the second image generation model. Specifically, in the above second image generation model, the input image to be processed can be multiplied by the mask image first to remove the original image in the area to be completed in the image to be processed, so as to obtain an image to be completed, and then using this image to be completed as a guiding condition to guide the above second image generation model to denoise the target noise image, so as to obtain a second completed image. Among them, the target noise image can be a random noise image, or can be obtained by adding noise to the image to be completed.

[0117] S1032. Fuse the first completed image and the second completed image to obtain a target completed image.

[0118] Specifically, before performing the above S1032, in order to obtain the above first completed image and second completed image, the above image processing device may execute the above S102 and the above S1031 in parallel, or may first execute the above S102 and then execute the above S1031, or may first execute the above S1031 and then execute the above S102.

[0119] In some specific embodiments, the above S1032 may specifically include: inputting the first completed image and the second completed image into a fusion module (Fusion Module, FM), where in the fusion module, extracting the first completed image feature of the first completed image and the second completed image feature of the second completed image, and fusing the first completed image and the second completed image according to the first completed image feature and the second completed image feature to obtain a target completed image output by the fusion module. Among them, the fusion module may include a neural network, the first completed image feature may be a feature map of the first completed image, and the second completed image feature may be a feature map of the second completed image.

[0120] Specifically, in the fusion module, a neural network (such as a convolutional neural network) can be used to extract features from the first completed image and the second completed image respectively to obtain the first completed image feature of the first completed image and the second completed image feature of the second completed image, and then fuse the first completed image feature and the second completed image feature to obtain a fused image feature, and generate a fused image (i.e., the above target completed image) based on this fused image feature.

[0121] It should be noted that as Figures 3 to 6As shown, when the second image generation model trained by the diffusion model in this embodiment is used to complete the incomplete regions A11 / B11 of the above-mentioned images to be processed A1 / B1 to obtain the second completed images A5 / B5, although it can achieve better image completion effects than the second image generation model trained by the non-diffusion model in terms of structure restoration and texture generation, since the diffusion model is a text-to-image model, there will be great uncertainty in completing the structure. The completion effect of the first image generation model in terms of structure completion and semantic consistency will be better than that of the second image generation model.

[0122] Moreover, as Figure 5 and Figure 6 shown, in this embodiment, after obtaining the second completed images A5 / B5, by further fusing the second completed images A5 / B5 and the first completed images A4 / B4 to obtain the final image completion result (i.e., the target completed images A6 / B6), not only can the image completion result of the above-mentioned second image generation model be optimized to more reliably complete the image completion task, but also the final image completion result can simultaneously possess the characteristics of the first completed image and the second completed image. Therefore, the fusion of non-diffusion generation and diffusion generation is realized, so that the completed image can have no obvious distortion and good semantic consistency, thereby significantly improving the image completion effect.

[0123] In addition, it should be noted that the image processing method in this embodiment can solve the problem that related technologies (such as diffusion-based image completion methods and patch-based image completion methods) are limited by the extracted low-dimensional image features and are difficult to generate high-quality completed images in the case of large-scale masks, and can be better applied to the application scenario of large-scale masks to improve the economic and application value of image completion.

[0124] In some embodiments where the above S103 includes the above S1031 and the above S1032, as Figure 9 shown, the above S103 may specifically include:

[0125] S1033. Extract features from the first completed image to obtain completed image features.

[0126] Specifically, after obtaining the above first completed image, the above image processing device can use a trained image feature extraction model (such as a convolutional neural network) to extract the image features of the above first completed image (i.e., completed image features). Among them, the completed image features can be the feature maps of the first completed image.

[0127] S1034. Based on the completed image features, the image to be processed, and the mask image, use the second image generation model to complete the area to be completed in the image to be processed, obtaining the target completed image.

[0128] In some examples, the above S1034 may specifically include: inputting the completed image features, the image to be processed, and the mask image into the second image generation model. In the second image generation model, denoise the target noise image according to the completed image features, the image to be processed, and the mask image, obtaining the target completed image output by the second image generation model. Specifically, in the above second image generation model, first multiply the input image to be processed by the mask image to remove the original image in the area to be completed in the image to be processed, obtaining the image to be completed, and then use the image to be completed and the input completed image features as guiding conditions to guide the above second image generation model to denoise the target noise image, obtaining the target completed image. Among them, the target noise image may be a random noise image, or may be obtained by adding noise to the image to be completed.

[0129] Moreover, it can be understood that in this embodiment, by using the completed image features of the above first completed image to guide the diffusion model to generate the final completed image (i.e., the above target completed image), it can make the target completed image aligned with the first completed image in terms of structure and semantics, so as to improve the problem of poor image completion effect of the diffusion model in terms of structure and semantics, and ensure the high quality of the target completed image.

[0130] In some alternative embodiments, as Figure 10 shown, the above S1034 may be replaced by S1035 and S1036, where:

[0131] S1035. Based on the completed image features, the image to be processed, and the mask image, use the second image generation model to complete the area to be completed in the image to be processed, obtaining the second completed image.

[0132] In some examples, the above S1035 may specifically include: inputting the completed image feature, the image to be processed, and the mask image into the second image generation model. In the second image generation model, denoising the target noise image according to the completed image feature, the image to be processed, and the mask image to obtain the second completed image output by the second image generation model. Specifically, in the above second image generation model, the input image to be processed can be multiplied by the mask image first to remove the original image in the area to be completed in the image to be processed, so as to obtain the image to be completed. Then, taking the image to be completed and the input completed image feature as guiding conditions, guiding the second image generation model to perform denoising processing on the target noise image to obtain the second completed image. Among them, the target noise image can be a random noise image, or can be obtained by adding noise to the image to be completed.

[0133] Moreover, it can be understood that in this embodiment, by using the completed image feature of the above first completed image to guide the diffusion model to generate the second completed image, it can make the second completed image align with the first completed image in terms of structure and semantics, which is beneficial to improving the problem of poor image completion effect of the diffusion model in terms of structure and semantics, so as to ensure the high quality of the second completed image.

[0134] S1036. Fuse the first completed image and the second completed image to obtain the target completed image.

[0135] It should be noted that since S1036 in this embodiment is the same as S1032 above, the specific implementation manner of S1036 in this embodiment can refer to the specific implementation manner of S1032 above, so it will not be elaborated here.

[0136] In the above embodiment, as Figure 10 shown, after obtaining the above second completed image and before fusing the above first completed image and the second completed image to obtain the target completed image, the above image processing method may further include:

[0137] S1037. Determine whether the second completed image meets the requirements.

[0138] Specifically, after obtaining the above second completed image, the above image processing device can send the above second completed image to the target object (for example, the user terminal). Then, the target object can receive the above second completed image and display the above second completed image in the interaction interface so that the user of the target object can view the above second completed image. Moreover, after the user of the target object views the above second completed image, the user can send the second feedback information for the above second completed image through the above interaction interface.

[0139] Among them, the second feedback information can be used to indicate whether the above-mentioned second completed image meets the requirements. Specifically, when the user believes that the above-mentioned second completed image can meet their needs, the user can send, through the target object, the second feedback information indicating that the above-mentioned second completed image meets the requirements to the above-mentioned image processing device; when the user believes that the above-mentioned second completed image cannot meet their needs, the user can send, through the target object, the second feedback information indicating that the above-mentioned second completed image does not meet the requirements to the above-mentioned image processing device.

[0140] Moreover, the above-mentioned image processing device can receive the second feedback information of the target object on the above-mentioned second completed image, and in response to the second feedback information indicating that the above-mentioned second completed image does not meet the requirements, determine that the above-mentioned second completed image does not meet the requirements, and in response to the second feedback information indicating that the above-mentioned second completed image meets the requirements, determine that the above-mentioned second completed image meets the requirements.

[0141] Correspondingly, the above-mentioned S1032 and the above-mentioned S1036 can specifically include: in response to the second completed image not meeting the requirements, fusing the first completed image and the second completed image to obtain a target completed image. Thus, the quality of the final image completion result can be further improved to solve the problem that the image completion result of the above-mentioned second image generation model (i.e., the above-mentioned second completed image) cannot meet the user's needs.

[0142] Moreover, when it is determined that the above-mentioned second completed image meets the requirements, the above-mentioned image processing device can not perform further processing, that is, the above-mentioned second completed image can be used as the final completion result without further fusing the first completed image and the second completed image to obtain a target completed image. Therefore, the completed images (i.e., the above-mentioned first completed image and the second completed image) can be returned to the user in stages, comprehensively considering the two factors of completion quality and running delay, giving the user a better user experience, and also facilitating the user to perform controllable, high-quality, and intelligent editing operations on the image, making the album use more intelligent and user-friendly.

[0143] As can be seen from the above, the image processing method provided in this embodiment obtains a to-be-processed image and a mask image. The mask image is used to identify the region to be filled in the to-be-processed image, and based on the to-be-processed image and the mask image, a first image generation model is used to perform filling processing on the region to be filled in the to-be-processed image to obtain a first filled image. The first image generation model is trained by a non-diffusion model. Then, based on the first filled image, the to-be-processed image, and the mask image, a second image generation model is used to perform filling processing on the region to be filled in the to-be-processed image to obtain a target filled image. The second image generation model is trained by a diffusion model. Thus, it realizes generating a good image filling result in terms of filling structure and semantic consistency using the non-diffusion model first, and then using this image filling result to guide the diffusion model to generate a good image filling result in terms of filling structure, structure restoration, texture generation, and semantic consistency, etc. It can give full play to the respective advantages of the non-diffusion model and the diffusion model to obtain a very good image filling effect in terms of filling structure, structure restoration, texture generation, and semantic consistency, etc., making the filled image have no obvious distortion and good semantic consistency, and thus can significantly improve the image filling effect.

[0144] Based on the method described in the above embodiment, this embodiment will be further described from the perspective of an image processing device. Please refer to Figure 11 , Figure 11 Specifically describes the image processing device provided in the embodiments of the present application. The image processing device includes: an acquisition module 301, a first filling module 302, and a second filling module 303, where:

[0145] (1) Acquisition module 301

[0146] The acquisition module 301 is used to acquire a to-be-processed image and a mask image. The mask image is used to identify the region to be filled in the to-be-processed image.

[0147] In some embodiments, the above acquisition module 301 may specifically include:

[0148] An acquisition unit for acquiring a to-be-processed image;

[0149] A determination unit for, in response to a region selection operation on the to-be-processed image, determining the region to be filled selected by the region selection operation from the to-be-processed image, and generating a mask image according to the region to be filled.

[0150] (2) First filling module 302

[0151] The first filling module 302 is used to, based on the to-be-processed image and the mask image, use a first image generation model to perform filling processing on the region to be filled in the to-be-processed image to obtain a first filled image. The first image generation model is trained by a non-diffusion model.

[0152] In some embodiments, the above-mentioned first completion module 302 may be specifically configured to:

[0153] Input the image to be processed and the mask image into the first image generation model. In the first image generation model, according to the image to be processed and the mask image, use a convolutional neural network, a transformer, and / or a generative adversarial network to perform completion processing on the area to be completed of the image to be processed, and obtain a first completed image output by the first image generation model.

[0154] (3) Second completion module 303

[0155] The second completion module 303 is used to perform completion processing on the area to be completed of the image to be processed according to the first completed image, the image to be processed, and the mask image by using a second image generation model, and obtain a target completed image. The second image generation model is trained by a diffusion model.

[0156] In some embodiments, the above-mentioned image processing device may further include:

[0157] (4) First determination module

[0158] The first determination module is used to determine whether the first completed image meets the requirements.

[0159] Correspondingly, the above-mentioned second completion module 303 may be specifically configured to:

[0160] In response to the first completed image not meeting the requirements, perform completion processing on the area to be completed of the image to be processed according to the first completed image, the image to be processed, and the mask image by using a second image generation model, and obtain a target completed image.

[0161] In some embodiments, the above-mentioned second completion module 303 may specifically include:

[0162] A feature extraction unit, configured to extract features from the first completed image to obtain completed image features;

[0163] A first completion unit, configured to perform completion processing on the area to be completed of the image to be processed according to the completed image features, the image to be processed, and the mask image by using a second image generation model, and obtain a target completed image.

[0164] In some specific embodiments, the above-mentioned first completion unit may be specifically configured to:

[0165] Input the completed image features, the image to be processed, and the mask image into the second image generation model. In the second image generation model, perform denoising processing on the target noise image according to the completed image features, the image to be processed, and the mask image, and obtain a target completed image output by the second image generation model.

[0166] In some other specific embodiments, the above-mentioned first completion unit may specifically include:

[0167] A completion subunit, configured to perform a completion process on a region to be completed of the image to be processed by using a second image generation model according to the completion image feature, the image to be processed, and the mask image, so as to obtain a second completed image;

[0168] A fusion subunit, configured to fuse the first completed image and the second completed image to obtain a target completed image.

[0169] In some other embodiments, the above-mentioned second completion module 303 may specifically include:

[0170] A second completion unit, configured to perform a completion process on a region to be completed of the image to be processed by using a second image generation model according to the image to be processed and the mask image, so as to obtain a second completed image;

[0171] A fusion unit, configured to fuse the first completed image and the second completed image to obtain a target completed image.

[0172] In some specific embodiments, the above-mentioned second completion unit may be specifically configured to:

[0173] Input the image to be processed and the mask image into the second image generation model, and in the second image generation model, perform a denoising process on the target noise image according to the image to be processed and the mask image, so as to obtain the second completed image output by the second image generation model.

[0174] In some embodiments, the above-mentioned image processing device may further include:

[0175] (5) A second determination module

[0176] The second determination module is configured to determine whether the second completed image meets the requirements.

[0177] Correspondingly, the above-mentioned fusing the first completed image and the second completed image to obtain a target completed image may specifically include:

[0178] In response to the second completed image not meeting the requirements, fuse the first completed image and the second completed image to obtain a target completed image.

[0179] In some specific embodiments, when the above-mentioned image processing device fuses the first completed image and the second completed image to obtain a target completed image, it may specifically execute:

[0180] Input the first completed image and the second completed image into the fusion module. In the fusion module, extract the first completed image feature of the first completed image and the second completed image feature of the second completed image, and generate a target completed image according to the first completed image feature and the second completed image feature.

[0181] In specific implementation, each of the above modules, units, and subunits can be implemented as an independent entity, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of each of the above modules, units, and subunits, reference can be made to the foregoing method embodiments and will not be elaborated herein.

[0182] As can be seen from the above, the image processing apparatus provided in this embodiment includes an acquisition module for acquiring an image to be processed and a mask image, where the mask image is used to identify the area to be completed in the image to be processed; a first completion module for completing the area to be completed in the image to be processed according to the image to be processed and the mask image by using a first image generation model to obtain a first completed image, and the first image generation model is trained by a non-diffusion model; a second completion module for completing the area to be completed in the image to be processed according to the first completed image, the image to be processed, and the mask image by using a second image generation model to obtain a target completed image, and the second image generation model is trained by a diffusion model. Thus, the image completion result of the non-diffusion model (such as a convolutional neural network) can be used to guide the diffusion model to generate an image completion result that is good in terms of completion structure, structure restoration, texture generation, and semantic consistency, so as to significantly improve the image completion effect.

[0183] Correspondingly, an embodiment of the present application further provides an electronic device, which can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a notebook computer, a touch screen, a game console, a personal computer, a personal digital assistant (PDA), etc. As Figure 12 shown, Figure 12 is a schematic structural diagram of the electronic device provided in the embodiment of the present application. The electronic device 400 includes a processor 401 having one or more processing cores, a memory 402 having one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. Among them, the processor 401 is electrically connected to the memory 402. Those skilled in the art can understand that the structural diagram of the electronic device shown in the figure does not constitute a limitation on the electronic device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0184] The processor 401 is the control center of the electronic device 400, connecting various parts of the entire electronic device 400 through various interfaces and circuits. By running or loading software programs and / or modules stored in the memory 402, and invoking the data stored in the memory 402, it executes various functions of the electronic device 400 and processes data.

[0185] In the embodiment of the present application, the processor 401 in the electronic device 400 will load the instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 will run the application programs stored in the memory 402 to implement various functions:

[0186] Obtain the image to be processed and the mask image, where the mask image is used to identify the area to be filled in the image to be processed;

[0187] According to the image to be processed and the mask image, use the first image generation model to perform filling processing on the area to be filled in the image to be processed to obtain the first filled image, and the first image generation model is trained by a non-diffusion model;

[0188] According to the first filled image, the image to be processed and the mask image, use the second image generation model to perform filling processing on the area to be filled in the image to be processed to obtain the target filled image, and the second image generation model is trained by a diffusion model.

[0189] For the specific implementation of each of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0190] Optionally, as Figure 12 shown, the electronic device 400 further includes: a touch display screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. Among them, the processor 401 is electrically connected to the touch display screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407 respectively. Those skilled in the art can understand that Figure 12 the structure of the electronic device shown in

[0191] The touch display screen 403 can be used to display a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface. The touch display screen 403 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. The touch panel can be used to collect touch operations of the user on or near it (such as operations of the user using any suitable object or accessory such as a finger or a stylus on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute the corresponding program. Optionally, the touch panel can include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch orientation of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 401, and can receive and execute the commands sent by the processor 401. The touch panel can cover the display panel. After the touch panel detects a touch operation on or near it, it is transmitted to the processor 401 to determine the type of touch event. Subsequently, the processor 401 provides a corresponding visual output on the display panel according to the type of touch event. In the embodiments of the present application, the touch panel and the display panel can be integrated into the touch display screen 403 to implement input and output functions. However, in some embodiments, the touch panel and the touch panel can be implemented as two independent components to implement input and output functions. That is, the touch display screen 403 can also be used as a part of the input unit 406 to implement the input function.

[0192] In the embodiments of the present application, by executing a game application program through the processor 401, a picture of a virtual three-dimensional scene is generated on the touch display screen 403. The picture includes a graphical user interface (UI interface), and the graphical user interface includes a second spatial orientation indicator, and a spatial orientation identifier corresponding to the target object is displayed on the second spatial orientation indicator. The spatial orientation identifier is used to indicate the orientation where the target object is located.

[0193] The touch display screen 403 can be used to present a picture of a virtual three-dimensional scene, as well as a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface.

[0194] The radio frequency circuit 404 can be used to transmit and receive radio frequency signals to establish wireless communication with a network device or other electronic devices through wireless communication, and transmit and receive signals with the network device or other electronic devices.

[0195] The audio circuit 405 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 405 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output. On the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 405, converted into audio data, and then the audio data is output to the processor 401 for processing. After that, it is sent to another electronic device through the radio frequency circuit 404, or the audio data is output to the memory 402 for further processing. The audio circuit 405 may also include an earphone jack to provide communication between the peripheral earphone and the electronic device.

[0196] The input unit 406 can be used to receive input digital, character information or feature information (such as fingerprint, iris, face information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0197] The power supply 407 is used to supply power to each component of the electronic device 400. Optionally, the power supply 407 can be logically connected to the processor 401 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 407 may also include one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, a power status indicator, and any other components.

[0198] Although Figure 12 not shown in the figure, the electronic device 400 may also include a camera, a sensor, a Wi-Fi module, a Bluetooth module, etc., which will not be elaborated here.

[0199] In the above embodiments, the descriptions of each embodiment have their own focuses. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0200] As can be seen from the above, the electronic device provided in this embodiment can improve the image completion effect.

[0201] Those of ordinary skill in the art can understand that all or part of the steps in the above various methods can be completed by instructions, or by controlling relevant hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0202] For this reason, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. These computer programs can be loaded by a processor to execute the steps in any one of the image processing methods provided by the embodiments of the present application. For example, the computer program can execute the following steps:

[0203] Obtain an image to be processed and a mask image, where the mask image is used to identify the area to be completed in the image to be processed;

[0204] According to the image to be processed and the mask image, use a first image generation model to complete the area to be completed in the image to be processed, and obtain a first completed image. The first image generation model is trained by a non-diffusion model;

[0205] According to the first completed image, the image to be processed and the mask image, use a second image generation model to complete the area to be completed in the image to be processed, and obtain a target completed image. The second image generation model is trained by a diffusion model.

[0206] For the specific implementation of each of the above operations, reference may be made to the previous embodiments, which will not be elaborated herein.

[0207] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0208] Since the computer program stored in the computer-readable storage medium can execute the steps in any of the image processing methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the image processing methods provided in the embodiments of the present application can be realized. For details, reference may be made to the previous embodiments, which will not be elaborated herein.

[0209] The above has introduced in detail an image processing method, device, computer-readable storage medium and electronic device provided by the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. An image processing method, characterized in that, Including: Obtain a to-be-processed image and a mask image, where the mask image is used to identify the to-be-completed area of the to-be-processed image; According to the to-be-processed image and the mask image, use a first image generation model to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a first completed image, where the first image generation model is trained by a non-diffusion model; According to the first completed image, the to-be-processed image, and the mask image, use a second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a target completed image, where the second image generation model is trained by a diffusion model.

2. The image processing method according to claim 1, characterized in that, After the step of using the first image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the to-be-processed image and the mask image to obtain a first completed image, the method further includes: Determine whether the first completed image meets the requirements; The step of using the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the first completed image, the to-be-processed image, and the mask image to obtain a target completed image specifically includes: In response to the first completed image not meeting the requirements, use the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the first completed image, the to-be-processed image, and the mask image to obtain a target completed image.

3. The image processing method according to claim 1, characterized in that, The step of using the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the first completed image, the to-be-processed image, and the mask image to obtain a target completed image specifically includes: Extract features from the first completed image to obtain completion image features; According to the completion image features, the to-be-processed image, and the mask image, use the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a target completed image.

4. The image processing method according to claim 3, characterized in that, The step of using the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the completion image features, the to-be-processed image, and the mask image to obtain a target completed image specifically includes: Input the completion image features, the to-be-processed image, and the mask image into the second image generation model. In the second image generation model, perform denoising processing on a target noise image according to the completion image features, the to-be-processed image, and the mask image to obtain the target completed image output by the second image generation model.

5. The image processing method according to claim 3, characterized in that, The step of using the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image according to the completion image features, the to-be-processed image, and the mask image to obtain a target completed image specifically includes: According to the completion image features, the to-be-processed image, and the mask image, use the second image generation model to perform a completion process on the to-be-completed area of the to-be-processed image to obtain a second completed image; Fuse the first completed image and the second completed image to obtain a target completed image.

6. The image processing method according to claim 1, characterized in that, Performing inpainting processing on the to-be-inpainted region of the to-be-processed image by using a second image generation model according to the first inpainted image, the to-be-processed image, and the mask image, to obtain a target inpainted image, specifically including: Performing inpainting processing on the to-be-inpainted region of the to-be-processed image by using a second image generation model according to the to-be-processed image and the mask image, to obtain a second inpainted image; Fusing the first inpainted image and the second inpainted image to obtain a target inpainted image.

7. The image processing method according to claim 6, characterized in that, The performing inpainting processing on the to-be-inpainted region of the to-be-processed image by using a second image generation model according to the to-be-processed image and the mask image, to obtain a second inpainted image, specifically including: Inputting the to-be-processed image and the mask image into a second image generation model, and in the second image generation model, performing denoising processing on a target noise image according to the to-be-processed image and the mask image, to obtain the second inpainted image output by the second image generation model.

8. The image processing method according to any one of claims 5 to 7, characterized in that, The fusing the first inpainted image and the second inpainted image to obtain a target inpainted image, specifically including: Inputting the first inpainted image and the second inpainted image into a fusion module, and in the fusion module, extracting a first inpainted image feature of the first inpainted image and a second inpainted image feature of the second inpainted image, and generating a target inpainted image according to the first inpainted image feature and the second inpainted image feature.

9. The image processing method according to any one of claims 5 to 7, characterized in that, After obtaining the second inpainted image, the method further includes: Determining whether the second inpainted image meets the requirements; The fusing the first inpainted image and the second inpainted image to obtain a target inpainted image, specifically including: In response to the second inpainted image not meeting the requirements, fusing the first inpainted image and the second inpainted image to obtain a target inpainted image.

10. The image processing method according to claim 1, characterized in that, The performing inpainting processing on the to-be-inpainted region of the to-be-processed image by using a first image generation model according to the to-be-processed image and the mask image, to obtain a first inpainted image, specifically including: Inputting the to-be-processed image and the mask image into a first image generation model, and in the first image generation model, performing inpainting processing on the to-be-inpainted region of the to-be-processed image by using a convolutional neural network, a transformer, and / or a generative adversarial network according to the to-be-processed image and the mask image, to obtain the first inpainted image output by the first image generation model.

11. The image processing method according to claim 1, wherein The obtaining the to-be-processed image and the mask image specifically includes: Obtaining a to-be-processed image; In response to a region selection operation on the to-be-processed image, determining the to-be-inpainted region selected by the region selection operation from the to-be-processed image, and generating a mask image according to the to-be-inpainted region.

12. An image processing apparatus, wherein Including: An obtaining module, configured to obtain a to-be-processed image and a mask image, where the mask image is used to identify the to-be-inpainted region of the to-be-processed image; The first completion module is configured to perform a completion process on the area to be completed of the image to be processed according to the image to be processed and the mask image by using a first image generation model, so as to obtain a first completed image, and the first image generation model is trained by a non-diffusion model; The second completion module is configured to perform a completion process on the area to be completed of the image to be processed according to the first completed image, the image to be processed and the mask image by using a second image generation model, so as to obtain a target completed image, and the second image generation model is trained by a diffusion model.

13. A computer-readable storage medium, wherein The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image processing method according to any one of claims 1-11.

14. An electronic device, wherein The electronic device includes a memory and a processor. A computer program is stored in the memory, and the processor executes the image processing method according to any one of claims 1-11 by calling the computer program stored in the memory.

Images