Image editing method, device, electronic device and readable storage medium

Abstract

The present application discloses an image editing method, device, electronic device and readable storage medium, wherein the image editing method comprises: obtaining an image to be edited and a corresponding image diffusion model; extracting features of the image to be edited by the image diffusion model to obtain latent space image features of the image to be edited; iteratively diffusing the image to be edited according to the latent space image features to obtain a latent space feature image; and image editing the latent space feature image by the image diffusion model to obtain a target image. The present application solves the technical problem of poor generalization of image editing.

Description

Technical Field

[0001] The present application relates to the field of image editing technology, and in particular to an image editing method, device, electronic device and readable storage medium. Background Art

[0002] With the continuous development of science and technology, the requirements for image editing capabilities are also increasing. Different from the traditional image editing method of modifying image content such as color, contrast and adding filters, deep learning technology is often combined with image editing to meet users' diverse and intelligent needs for image editing.

[0003] Currently, image editing is usually implemented based on generative adversarial networks, that is, the image is edited by editing the representation of the image in the latent space of the GAN model. However, since the GAN model can only edit images of specific categories, when users want to edit natural images, they cannot achieve the editing effect that meets the user's needs, which makes it easy for users to have a poor image editing experience. Therefore, the current image editing has poor generalization. Summary of the invention

[0004] The main purpose of the present application is to provide an image editing method, device, electronic device and readable storage medium, aiming to solve the technical problem of poor editing generalization in image editing in the prior art.

[0005] To achieve the above object, the present application provides an image editing method, the image editing method comprising:

[0006] Obtaining the image to be edited and the corresponding image diffusion model;

[0007] Extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited;

[0008] According to the latent space image features, iteratively diffuse the image to be edited to obtain a latent space feature image;

[0009] The latent space feature image is edited by using the image diffusion model to obtain a target image.

[0010] Optionally, the image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited.

[0011] The step of extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited comprises:

[0012] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate natural image features of the natural image to be edited in a latent space;

[0013] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate reference image features of the natural image to be edited in a latent space;

[0014] The natural image features and the reference image features are used together as the latent space image features.

[0015] Optionally, before the step of iteratively diffusing the image to be edited according to the selection identifier and according to the latent space image feature to obtain a latent space feature image, the image editing method further includes:

[0016] Acquire first intermediate sampling information of the natural image to be edited during an iterative inverse sampling process and second intermediate sampling information of the reference image to be edited during an iterative inverse sampling process;

[0017] Converting the first intermediate sampling information into a first intermediate sampling feature, and converting the second intermediate sampling information into a second intermediate sampling feature;

[0018] Calculating a target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature;

[0019] According to the target guided gradient and the latent space image feature, the image to be edited is iteratively diffused to obtain a latent space feature image.

[0020] Optionally, the step of calculating the target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature includes:

[0021] Calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing according to the first intermediate sampling feature and the second intermediate sampling feature;

[0022] According to the energy loss function corresponding to the edit similarity, the target guided gradient of the latent space image feature is calculated.

[0023] Optionally, the energy loss function is as follows:

[0024]

[0025] Where S is the editing similarity, F t gen is the first intermediate sampling feature, F t gudis the second intermediate sampling feature, m gen is the target editing area before editing, m gud The target editing area after editing.

[0026] Optionally, the target guidance gradient is calculated as follows:

[0027]

[0028]

[0029] Optionally, the target image includes a first target image and a second target image, wherein the first target image carries image features of the natural image to be edited, and the second target image carries image features of the natural image to be edited and the reference image to be edited.

[0030] To achieve the above object, the present application also provides an image editing device, the image editing device comprising:

[0031] An acquisition module, used for acquiring an image to be edited and a corresponding image diffusion model;

[0032] An extraction module, used for extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited;

[0033] A diffusion module, used for iteratively diffusing the image to be edited according to the latent space image features to obtain a latent space feature image;

[0034] The editing module is used to perform image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0035] Optionally, the image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited, and the extraction module is further used for:

[0036] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate natural image features of the natural image to be edited in a latent space;

[0037] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate reference image features of the natural image to be edited in a latent space;

[0038] The natural image features and the reference image features are used together as the latent space image features.

[0039] Optionally, the image editing device is further used for:

[0040] Acquire first intermediate sampling information of the natural image to be edited during an iterative inverse sampling process and second intermediate sampling information of the reference image to be edited during an iterative inverse sampling process;

[0041] Converting the first intermediate sampling information into a first intermediate sampling feature, and converting the second intermediate sampling information into a second intermediate sampling feature;

[0042] Calculating a target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature;

[0043] According to the target guided gradient and the latent space image feature, the image to be edited is iteratively diffused to obtain a latent space feature image.

[0044] Optionally, the image editing device is further used for:

[0045] Calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing according to the first intermediate sampling feature and the second intermediate sampling feature;

[0046] According to the energy loss function corresponding to the edit similarity, the target guided gradient of the latent space image feature is calculated.

[0047] Optionally, the energy loss function is as follows:

[0048]

[0049] Where S is the editing similarity, F t gen is the first intermediate sampling feature, F t gud is the second intermediate sampling feature, m gen is the target editing area before editing, m gud The target editing area after editing.

[0050] Optionally, the target guidance gradient is calculated as follows:

[0051]

[0052]

[0053] Optionally, the target image includes a first target image and a second target image, wherein the first target image carries image features of the natural image to be edited, and the second target image carries image features of the natural image to be edited and the reference image to be edited.

[0054] The present application also provides an electronic device, comprising: at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can perform the steps of the image editing method as described above.

[0055] The present application also provides a computer-readable storage medium, on which is stored a program for implementing the image editing method. When the program of the image editing method is executed by a processor, the steps of the image editing method as described above are implemented.

[0056] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image editing method when executed by a processor.

[0057] The present application provides an image editing method, device, electronic device and readable storage medium, that is, obtaining an image to be edited and a corresponding image diffusion model; performing feature extraction on the image to be edited through the image diffusion model to obtain latent space image features of the image to be edited; iteratively diffusing the image to be edited according to the latent space image features to obtain a latent space feature image; and performing image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0058] When performing image editing of an image to be edited, the present application first obtains the image to be edited and its corresponding image diffusion model, and then uses the image diffusion model to perform feature extraction on the image to be edited to extract the latent space image features of the image to be edited in the latent space, and then iteratively diffuses the image to be edited through the latent space image features to obtain the latent space feature image, and finally uses the image diffusion model to complete the image editing of the latent space feature image to obtain the target image. Since the image diffusion model can ultimately output a target image that meets user needs depending on the image to be edited, and at the same time, the image diffusion model itself has a strong generalization ability, and then when inputting the image to be edited for image editing, there is no need to limit the image type of the image to be edited, and thus the purpose of editing any image to be edited to obtain a target image that meets user needs can be achieved.

[0059] After extracting the latent space image features of the image to be edited, the image to be edited can be iteratively diffused based on the latent space image features to obtain a latent space feature image under the latent space, thereby achieving the purpose of arbitrarily editing the image to be edited. Finally, the image editing of the latent space feature image by the image diffusion model can obtain a target image that meets the user's editing needs. That is, the purpose of editing any image into a target image that meets the user's editing needs is achieved with the help of the powerful generalization ability of the image diffusion model.

[0060] Based on this, the present application uses an image diffusion model to extract features of the image to be edited, and then obtains the latent space image features of the image to be edited in the latent space, and relies on the latent space image features to complete the iterative diffusion of the image to be edited, and obtains the latent space feature image, and finally uses the image diffusion model to complete the image editing of the latent space feature image, and obtains the target image, thereby realizing the use of the powerful generalization ability of the image diffusion model to complete the purpose of editing the image to be edited. Instead of relying on the generative adversarial network for image editing when performing image editing. That is, it overcomes the technical defect that the GAN model can only edit images of a specific category, resulting in the user's inability to achieve the editing effect that meets the user's needs when the user wants to edit the natural image, which makes it easy for the user to have a poor image editing experience, so the editing generalization of image editing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0063] Figure 1 A flowchart of an image editing method provided in Example 1 of the present application;

[0064] Figure 2 A schematic diagram of an editing process for performing image editing based on an image diffusion model in an image editing method provided in Embodiment 1 of the present application;

[0065] Figure 3 A schematic diagram of the structure of an image editing device provided in Embodiment 2 of the present application;

[0066] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Example 3 of the present application.

[0067] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0069] Embodiment 1

[0070] The present application embodiment provides an image editing method. In the first embodiment of the image editing method of the present application, referring to Figure 1 , the image editing method comprises:

[0071] Step S10, obtaining the image to be edited and the corresponding image diffusion model;

[0072] Step S20, extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited;

[0073] Step S30, iteratively diffusing the image to be edited according to the latent space image features to obtain a latent space feature image;

[0074] Step S40, performing image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0075] In this embodiment, it should be noted that although Figure 1 The logical sequence is shown, but in some cases, the steps shown or described can be performed in a different order from that here. The image editing device for deploying the image editing method can be specifically a computer, a mobile phone or a personal PC, etc. The image to be edited is used to characterize the image waiting for image editing. The embodiment of the present application does not specifically limit the image content carried by the image to be edited, that is, it does not limit the image category to which the image to be edited belongs. The image to be edited can be either a person image or an animal image. The image diffusion model is used to realize image editing of the image to be edited, that is, a new image similar to the training data can be generated. It can be understood that the image diffusion model is a pre-trained diffusion model. The larger the amount of training data, the stronger the generalization ability of the image diffusion model. The image diffusion model can specifically include an autoencoder, an autodecoder, an inverse sampling unit, a sampling unit, a diffusion unit, and a denoising unit. Before extracting the features of the image to be edited through the image diffusion model, it is necessary to encode the image to be edited into a latent space through the autoencoder of the image diffusion model, and then generate the latent space image features of the image to be edited in the inverse sampling process of the image diffusion model, wherein the latent space image features can be specifically represented in the form of a matrix or a vector.

[0076] In addition, it should be noted that the latent space feature image is used to characterize the feature image under the latent space. After finding the latent space image features of an image to be edited through the inverse sampling process of the image diffusion model, the original image can be further reconstructed through the forward sampling process of the image diffusion model. The inverse sampling process and the forward sampling process can be completed by iterating T steps. For example, in an implementable manner, the image to be edited and the pre-trained image diffusion model ε can be first selected. θ , and then the image to be edited is encoded into the latent space through the autoencoder of the image diffusion model, and then the image is encoded into the latent space through the image diffusion model ε θ Perform the inverse sampling process to find the latent space image feature Z of the image to be edited t , at each diffusion time step t, the latent space image feature Z is iteratively updated t , and obtain the latent space feature image. Finally, after the iterative diffusion is completed, the encoder of the image diffusion model decodes the edited image from the latent space to obtain the target image, where the target image is used to represent the image that meets the user's editing requirements.

[0077] In addition, it should be noted that since the image generation process of the image diffusion model includes T-step sampling, the latent space of the image diffusion model is discrete, which may include Z T , ... Z 0 , and then the drag-and-drop image editing operation is modeled as a change in feature correspondence, that is, the image feature is transformed from the position before editing to the position after editing, so that the pre-trained image diffusion model can be used without any training to achieve image editing that meets the user's editing needs.

[0078] As an example, step S10 to step S40 include: obtaining an image to be edited and completing a pre-trained image diffusion model; encoding the image to be edited into a latent space through an autoencoder of the image diffusion model, and inversely sampling the image to be edited located in the latent space through the image diffusion model to obtain latent space image features of the image to be edited; at each time step, iteratively diffusing the image to be edited through the image diffusion model according to the latent space image features to obtain a latent space feature image; decoding the latent space feature image through the autodecoder of the image diffusion model to obtain a target image.

[0079] The embodiment of the present application first obtains the image to be edited and the pre-trained image diffusion model, and then extracts the features of the image to be edited through the image diffusion model to obtain the latent space image features of the image to be edited, and finally decodes the image to be edited into a target image after iterative diffusion of a preset time step based on the image diffusion model. Since the image diffusion model has a strong generalization ability, it can achieve the purpose of image editing that meets the user's editing needs for any image to be edited, rather than relying on the generative adversarial network to perform image editing when performing image editing. Therefore, it overcomes the technical defect that the GAN model can only edit images of a specific category, resulting in the user being unable to achieve an editing effect that meets the user's needs when the user wants to edit a natural image, which makes it easy for the user to have a poor image editing experience. Therefore, the editing generalization of image editing is improved.

[0080] The image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited, and the step of extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited includes:

[0081] Step A10, performing iterative inverse sampling on the natural image to be edited by using the image diffusion model to generate natural image features of the natural image to be edited in a latent space;

[0082] Step A20, iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate reference image features of the natural image to be edited in a latent space;

[0083] Step A30: taking the natural image features and the reference image features together as the latent space image features.

[0084] In this embodiment, it should be noted that, in the process of image editing, the user may need to edit a target image that carries image elements that are not present in the original image to be edited. For example, assuming that there are apples in image A and bananas in image B, and the user's editing requirement is to have both apples and bananas in the target image, it is necessary to prepare reference images at the beginning of image editing, that is, the natural image to be edited is used to represent the natural image to be edited, and the reference image to be edited is used to represent the reference image of the natural image to be edited, wherein both the natural image to be edited and the reference image to be edited carry the image features required by the target image, and then in the process of editing the reference image, the reference image to be edited is used to represent the reference image of the natural image to be edited. When extracting features, feature extraction needs to be performed for different images. For example, in one feasible method, after selecting a natural image to be edited and a reference image to be edited of the natural image to be edited, the natural image to be edited and the reference image to be edited are encoded into a latent space through an autoencoder of an image diffusion model, thereby performing feature extraction on the natural image to be edited and the reference image to be edited in the latent space to obtain natural image features and reference image features, wherein the natural image features refer to the image features of the natural image to be edited in the latent space, and the reference image features refer to the image features of the reference image to be edited in the latent space, and finally both the natural image features and the reference image features are used as latent space image features.

[0085] As an example, steps A10 to A30 include: iteratively desampling the natural image to be edited through the image diffusion model to generate natural image features of the natural image to be edited in the latent space; iteratively desampling the natural image to be edited through the image diffusion model to generate reference image features of the natural image to be edited in the latent space; and using the natural image features and the reference image features together as the latent space image features.

[0086] Wherein, before the step of iteratively diffusing the image to be edited according to the latent space image features to obtain the latent space feature image, the image editing method further includes:

[0087] Step B10, obtaining first intermediate sampling information of the natural image to be edited during the iterative inverse sampling process and second intermediate sampling information of the reference image to be edited during the iterative inverse sampling process;

[0088] Step B20, converting the first intermediate sampling information into a first intermediate sampling feature, and converting the second intermediate sampling information into a second intermediate sampling feature;

[0089] Step B30, calculating the target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature;

[0090] Step B40, iteratively diffuse the image to be edited according to the target guided gradient and the latent space image feature to obtain a latent space feature image.

[0091] In this embodiment, it should be noted that a large number of intermediate features will be generated during the inverse sampling process of the natural image to be edited and the reference image to be edited. In order to make full use of the intermediate features in the inverse sampling process, the embodiment of the present application proposes to construct an intermediate storage of the inverse sampling process to reuse the intermediate information in the inverse sampling process and guide the subsequent editing process. It can be understood that in the sampling generation stage, the guidance information provided by the intermediate feature storage can be divided into the following two parts: 1) feature transfer, which is used to maintain the consistency between the target image and the image to be edited; 2) used to convert the editing operation into a gradient correction in the diffusion process. By reusing the guidance information of image editing in the inverse sampling process, the additional computational overhead can be saved while generating the guidance information. For example, in an implementable manner, after t steps of inverse sampling, the natural image feature Z of the natural image to be edited in the latent space is obtained. t , and construct intermediate storage, where {Z t gud , K t gud ,V t gud} is used to represent the first intermediate sampling information of the natural image to be edited, {Z t ref , K t ref ,V t ref} is used to represent the second intermediate sampling information of the reference image to be edited. If there is no reference image to be edited during the image editing process, then {Z t ref , K t ref ,V t ref} is empty, and the generated natural image feature Z t As the starting point, the edited image is generated after t steps of sampling. There are two guiding branches in this process: 1) {K t gud , V t gud} and {K t ref , V t ref Replace the image diffusion model ε at the current time step by feature transfer θThe purpose of this operation is to keep the final image editing result consistent with the original image to be edited; 2) Generate a guided gradient through the energy loss function, where m gud and m gen represents the position of the image content of the image to be edited before editing and the position of the image content of the image to be edited after editing. This constraint is performed on the feature domain. In the embodiment of the present application, Z is converted into t gud and Z t gen Transform to the feature domain to generate F t gud and F t gen , where F t gud is the first intermediate sampling feature, F t gen For the second intermediate sampling feature, the target guided gradient is used to measure the loss of the pre-editing region of the first intermediate sampling feature and the post-editing region of the second intermediate sampling feature.

[0092] As an example, step B10 to step B40 include: obtaining the first intermediate sampling information of the natural image to be edited in the iterative inverse sampling process and the second intermediate sampling information of the reference image to be edited in the iterative inverse sampling process; converting the first intermediate sampling information to the feature domain to obtain the first intermediate sampling feature, and converting the second intermediate sampling information to the feature domain to obtain the second intermediate sampling feature; calculating the target guided gradient of the latent space image feature based on the first intermediate sampling feature and the second intermediate sampling feature; iteratively diffusing the image to be edited based on the target guided gradient and the latent space image feature to obtain a latent space feature image.

[0093] The step of calculating the target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature comprises:

[0094] Step C10, calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing according to the first intermediate sampling feature and the second intermediate sampling feature;

[0095] Step C20, calculating the target guided gradient of the latent space image feature according to the energy loss function corresponding to the edit similarity.

[0096] In this embodiment, it should be noted that the editing similarity is used to characterize the similarity of the regional similarity of the first intermediate sampling feature and the second intermediate sampling feature before and after editing. The energy loss function is constructed by reusing the image diffusion model, and the calculated target guided gradient is imported into the iterative diffusion process of the image diffusion model, and finally the latent space feature image is obtained, which can ensure the accuracy of image editing while avoiding the training overhead of the energy function, wherein the target editing area refers to the specified area in the image to be edited.

[0097] As an example, steps C10 to C20 include: calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing through the first intermediate sampling feature and the second intermediate sampling feature; calculating the target guided gradient of the latent space image feature through the energy loss function constructed by the editing similarity.

[0098] Wherein, the energy loss function is as follows:

[0099]

[0100] Where S is the editing similarity, F t gen is the first intermediate sampling feature, F t gud is the second intermediate sampling feature, m gen is the target editing area before editing, m gud The target editing area after editing.

[0101] The calculation formula of the target guidance gradient is as follows:

[0102]

[0103]

[0104] The target image includes a first target image and a second target image, wherein the first target image carries image features of the natural image to be edited, and the second target image carries image features of the natural image to be edited and the reference image to be edited.

[0105] It can be understood that the first target image can be a natural image obtained after image editing processing such as moving objects within the image, changing the size of the image, and dragging points of the image is performed on the natural image to be edited. The second target image can be a natural image after content is copied and pasted or the appearance is changed between the natural image to be edited and the reference image to be edited. When performing image editing, different image editing tasks can be performed through the image diffusion model depending on whether there is a reference image to be edited, thereby expanding the scene adaptability of image editing based on the image diffusion model.

[0106] In one practicable manner, referring to Figure 2 , Figure 2 Schematic diagram of the editing process of image editing based on the image diffusion model, wherein the natural image X to be edited and the reference image Y to be edited are input into the image diffusion model together. The image diffusion model performs an inverse sampling process to obtain latent space image features of different images, and stores the intermediate features in the inverse sampling process to guide the sampling process of the image diffusion model, and finally obtains the target image.

[0107] An embodiment of the present application provides an image editing method, namely, obtaining an image to be edited and a corresponding image diffusion model; performing feature extraction on the image to be edited through the image diffusion model to obtain latent space image features of the image to be edited; iteratively diffusing the image to be edited according to the latent space image features to obtain a latent space feature image; and performing image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0108] In the embodiment of the present application, when performing image editing of an image to be edited, the image to be edited and its corresponding image diffusion model are first obtained, and then feature extraction is performed on the image to be edited through the image diffusion model to extract the latent space image features of the image to be edited in the latent space, and then the image to be edited is iteratively diffused through the latent space image features to obtain the latent space feature image, and finally the image diffusion model completes the image editing of the latent space feature image to obtain the target image. Since the image diffusion model can rely on the image to be edited to finally output a target image that meets user needs, at the same time, the image diffusion model itself has a strong generalization ability, and then when the image to be edited is input for image editing, there is no need to limit the image type of the image to be edited, and thus the purpose of editing any image to be edited to obtain a target image that meets user needs can be achieved.

[0109] After extracting the latent space image features of the image to be edited, the image to be edited can be iteratively diffused based on the latent space image features to obtain a latent space feature image under the latent space, thereby achieving the purpose of arbitrarily editing the image to be edited. Finally, the image editing of the latent space feature image by the image diffusion model can obtain a target image that meets the user's editing needs. That is, the purpose of editing any image into a target image that meets the user's editing needs is achieved with the help of the powerful generalization ability of the image diffusion model.

[0110] Based on this, the embodiment of the present application uses an image diffusion model to extract features of the image to be edited, and then obtains the latent space image features of the image to be edited in the latent space, and relies on the latent space image features to complete the iterative diffusion of the image to be edited, and obtains the latent space feature image, and finally uses the image diffusion model to complete the image editing of the latent space feature image to obtain the target image, thereby realizing the use of the powerful generalization ability of the image diffusion model to complete the purpose of editing the image to be edited. Instead of relying on the generative adversarial network to perform image editing when performing image editing. That is, it overcomes the technical defect that the GAN model can only edit images of specific categories, resulting in the user's inability to achieve the editing effect that meets the user's needs when the user wants to edit the natural image, and thus easily leads to the situation where the user has a poor image editing experience, so the editing generalization of image editing is improved.

[0111] Embodiment 2

[0112] The present application also provides an image editing device, referring to Figure 3 , the image editing device comprises:

[0113] An acquisition module 101 is used to acquire an image to be edited and a corresponding image diffusion model;

[0114] An extraction module 102, configured to extract features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited;

[0115] A diffusion module 103, configured to iteratively diffuse the image to be edited according to the latent space image features to obtain a latent space feature image;

[0116] The editing module 104 is used to perform image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0117] Optionally, the image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited, and the extraction module 102 is further used for:

[0118] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate natural image features of the natural image to be edited in a latent space;

[0119] Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate reference image features of the natural image to be edited in a latent space;

[0120] The natural image features and the reference image features are used together as the latent space image features.

[0121] Optionally, the image editing device is further used for:

[0122] Acquire first intermediate sampling information of the natural image to be edited during an iterative inverse sampling process and second intermediate sampling information of the reference image to be edited during an iterative inverse sampling process;

[0123] Converting the first intermediate sampling information into a first intermediate sampling feature, and converting the second intermediate sampling information into a second intermediate sampling feature;

[0124] Calculating a target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature;

[0125] According to the target guided gradient and the latent space image feature, the image to be edited is iteratively diffused to obtain a latent space feature image.

[0126] Optionally, the image editing device is further used for:

[0127] Calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing according to the first intermediate sampling feature and the second intermediate sampling feature;

[0128] According to the energy loss function corresponding to the edit similarity, the target guided gradient of the latent space image feature is calculated.

[0129] Optionally, the energy loss function is as follows:

[0130]

[0131] Where S is the editing similarity, F t gen is the first intermediate sampling feature, F t gud is the second intermediate sampling feature, m gen is the target editing area before editing, m gud The target editing area after editing.

[0132] Optionally, the target guidance gradient is calculated as follows:

[0133]

[0134]

[0135] Optionally, the target image includes a first target image and a second target image, wherein the first target image carries image features of the natural image to be edited, and the second target image carries image features of the natural image to be edited and the reference image to be edited.

[0136] The image editing device provided by the present invention adopts the image editing method in the above embodiment to solve the technical problem of poor generalization of image editing. Compared with the prior art, the beneficial effects of the image editing device provided by the embodiment of the present invention are the same as the beneficial effects of the image editing method provided by the above embodiment, and other technical features in the image editing device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0137] Embodiment 3

[0138] An embodiment of the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image editing method in the above-mentioned embodiment one.

[0139] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 4 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus.

[0141] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0142] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0143] The electronic device provided by the present invention adopts the image editing method in the above embodiment to solve the technical problem of poor generalization of image editing. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present invention are the same as the beneficial effects of the image editing method provided by the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0144] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0146] Embodiment 4

[0147] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the image editing method in the above embodiment.

[0148] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0149] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0150] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains an image to be edited and a corresponding image diffusion model; extracts features of the image to be edited through the image diffusion model to obtain latent space image features of the image to be edited; iteratively diffuses the image to be edited according to the latent space image features to obtain a latent space feature image; and performs image editing on the latent space feature image through the image diffusion model to obtain a target image.

[0151] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0154] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned image editing method, which solves the technical problem of poor editing generalization in image editing. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the image editing method provided by the above-mentioned embodiment, and will not be described in detail here.

[0155] Embodiment 5

[0156] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image editing method when executed by a processor.

[0157] The computer program product provided by the present application solves the technical problem of poor generalization of image editing. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the image editing method provided by the above embodiment, which will not be repeated here.

[0158] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An image editing method, characterized in that: The image editing method comprises: Obtaining the image to be edited and the corresponding image diffusion model; Extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited; According to the latent space image features, iteratively diffuse the image to be edited to obtain a latent space feature image; Performing image editing on the latent space feature image through the image diffusion model to obtain a target image; The image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited, and the step of extracting features of the image to be edited by using the image diffusion model to obtain latent space image features of the image to be edited includes: Iteratively inverse sampling the natural image to be edited by using the image diffusion model to generate natural image features of the natural image to be edited in a latent space; Iteratively inverse sampling the reference image to be edited by using the image diffusion model to generate reference image features of the reference image to be edited in a latent space; Using the natural image features and the reference image features together as the latent space image features; Before the step of iteratively diffusing the image to be edited according to the latent space image features to obtain the latent space feature image, the image editing method further includes: Acquire first intermediate sampling information of the natural image to be edited during an iterative inverse sampling process and second intermediate sampling information of the reference image to be edited during an iterative inverse sampling process; Converting the first intermediate sampling information into a first intermediate sampling feature, and converting the second intermediate sampling information into a second intermediate sampling feature; Calculating a target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature; According to the target guided gradient and the latent space image feature, the image to be edited is iteratively diffused to obtain a latent space feature image.

2. The image editing method according to claim 1, characterized in that: The step of calculating the target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature comprises: Calculating the editing similarity of the target editing area of ​​the image to be edited before and after editing according to the first intermediate sampling feature and the second intermediate sampling feature; According to the energy loss function corresponding to the edit similarity, the target guided gradient of the latent space image feature is calculated.

3. The image editing method according to claim 2, characterized in that: The energy loss function is as follows: Where S is the editing similarity, F t gen is the first intermediate sampling feature, F t gud is the second intermediate sampling feature, m gen is the target editing area before editing, m gud The target editing area after editing.

4. The image editing method according to claim 2, characterized in that: The calculation formula of the target guided gradient is as follows: in, Guide the gradient for the target, is the intermediate guide gradient, is the initial guide gradient.

5. The image editing method according to claim 1, characterized in that: The target image includes a first target image and a second target image, wherein the first target image carries image features of the natural image to be edited, and the second target image carries image features of the natural image to be edited and the reference image to be edited.

6. An image editing device, characterized in that: The image editing device comprises: An acquisition module, used for acquiring an image to be edited and a corresponding image diffusion model; An extraction module is used to extract features of the image to be edited through the image diffusion model to obtain latent space image features of the image to be edited; the image to be edited includes a natural image to be edited and a reference image to be edited of the natural image to be edited, and the extraction module is also used to iteratively desample the natural image to be edited through the image diffusion model to generate natural image features of the natural image to be edited in latent space; iteratively desample the reference image to be edited through the image diffusion model to generate reference image features of the reference image to be edited in latent space; the natural image features and the reference image features are used together as the latent space image features A diffusion module is used to iteratively diffuse the image to be edited according to the latent space image feature to obtain a latent space feature image; the diffusion module is also used to obtain the first intermediate sampling information of the natural image to be edited during the iterative inverse sampling process and the second intermediate sampling information of the reference image to be edited during the iterative inverse sampling process; convert the first intermediate sampling information into a first intermediate sampling feature, and convert the second intermediate sampling information into a second intermediate sampling feature; calculate the target guided gradient of the latent space image feature according to the first intermediate sampling feature and the second intermediate sampling feature; iteratively diffuse the image to be edited according to the target guided gradient and the latent space image feature to obtain a latent space feature image; The editing module is used to perform image editing on the latent space feature image through the image diffusion model to obtain a target image.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the image editing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program for implementing the image editing method, and the program for implementing the image editing method is executed by a processor to implement the steps of the image editing method according to any one of claims 1 to 5.

Images