Image generation method and device

Abstract

The present invention provides an image generation method and apparatus, relating to the field of image processing. In this method, initial edge information of at least one object in an original image is extracted; an image editing operation is obtained, and based on the image editing operation, a target object to be edited and desired edge information of the target object are determined; the original image, the initial edge information of the target object, and the desired edge information of the target object are input into an image generation model, resulting in a target image output by the image generation model, which includes the target object with the desired edge information. This method can enhance the enjoyment of human-computer interaction with images.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and specifically, to an image generation method and device. Background Art

[0002] With the rapid development of deep learning technology and computer vision technology in recent years, computer vision technology has been widely used in many fields such as security monitoring, medical health, etc.

[0003] In some application scenarios, people hope to use computer vision technology to deform or edit objects of interest in images to generate new images, thereby increasing the fun of human-computer interaction operations on images. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide an image generation method and apparatus to at least solve the problem that users are currently unable to edit objects in images.

[0005] According to one aspect of an embodiment of the present application, an image generation method is provided, comprising: extracting initial edge information of at least one object in an original image; obtaining an image editing operation, and determining a target object to be edited and desired edge information of the target object based on the image editing operation; inputting the original image, the initial edge information of the target object, and the desired edge information of the target object into an image generation model, to obtain a target image output by the image generation model, wherein the target image includes the target object having the desired edge information.

[0006] According to one aspect of an embodiment of the present application, an image generation device is also provided, including: an initial edge information extraction unit, configured to extract initial edge information of at least one object in an original image; an editing edge information acquisition unit, configured to obtain an image editing operation, and determine a target object to be edited and desired edge information of the target object based on the image editing operation; a target image generation unit, configured to input the original image, the initial edge information of the target object, and the desired edge information of the target object into an image generation model, to obtain a target image output by the image generation model, wherein the target image includes the target object having the desired edge information.

[0007] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the image generation method as described above.

[0008] According to another aspect of an embodiment of the present application, a machine-readable storage medium is further provided, which stores executable instructions. When the instructions are executed, the machine executes the image generation method as described above.

[0009] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0010] The initial edge information of the object in the original image is extracted. User actions are then used to edit this initial edge information to determine the target object to be edited and the desired edge information for the target object. This initial edge information, the desired edge information, and the original image are then input into the image generation model, which then outputs a target image of the target object with the desired edge information. By leveraging the powerful generalization capabilities of the image generation model, users can edit the edge information of objects in the original image to obtain a target image that matches the desired state. This simplifies operations and increases the fun of human-computer interaction with images. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A further understanding of the nature and advantages of the embodiments of the present application can be achieved by referring to the following drawings. In the drawings, similar components or features may have the same reference numerals. The drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the application. Together with the following detailed description, they are used to explain the embodiments of the present application, but do not constitute a limitation of the embodiments of the present application. In the drawings:

[0012] Figure 1 A flowchart showing an example of an image generation method according to an embodiment of the present application is shown;

[0013] Figure 2 A flowchart illustrating an example of determining desired edge information in an image generation method according to an embodiment of the present application is shown;

[0014] Figure 3 A flowchart of an example of constructing a generative adversarial network with an image generation model according to an embodiment of the present application is shown;

[0015] Figure 4 A schematic diagram of the structure of an example of UNet is shown;

[0016] Figure 5 A schematic diagram showing the structure of an example of a self-attention network;

[0017] Figure 6 A flowchart of an example of training an image generation model according to an embodiment of the present application is shown;

[0018] Figure 7 A schematic diagram showing the effects of the image generation method according to an embodiment of the present application in some application scenarios;

[0019] Figure 8 A structural block diagram of an example of an image generating device according to an embodiment of the present application is shown;

[0020] Figure 9 A hardware structure diagram of an example of an electronic device for generating an image according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0023] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0024] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0025] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0027] In this article, the term "Generative Adversarial Networks" (GAN) refers to a deep learning model that produces reasonably good outputs through interactive learning between (at least) two modules in the framework: a generative model and a discriminative model.

[0028] Figure 1 The flowchart of an example of the image generation method according to the embodiment of the present application is shown. The execution subject of the method of the embodiment of the present application can be a terminal device such as a mobile phone or a tablet computer.

[0029] like Figure 1 In the illustrated process 100, initial edge information of at least one object in the original image is extracted in step 110. For example, the object may represent an object or a part of an object in the image, such as a person or a part of a person (e.g., a mouth, an arm, etc.) in the image.

[0030] Specifically, only an original image containing an object needs to be input (e.g., by a user) to automatically extract edge information of the object in the image. It should be understood that various extraction methods can be used to extract the corresponding edge information, such as using edge extraction algorithms in traditional computer vision or other methods to extract edge information.

[0031] In step 120, an image editing operation is obtained, and based on the image editing operation, a target object to be edited and desired edge information of the target object are determined. For example, a user can select edge information corresponding to an object of interest or a portion of an object in the image and move or deform it. For example, through user operation, the edge information corresponding to the "closed mouth state" of the "mouth" in the original image can be adjusted to the edge information corresponding to the "open mouth state." In this example, the "mouth" is the target object, the edge information corresponding to the "open mouth state" is the initial edge information, and the edge information corresponding to the "closed mouth state" is the desired edge information.

[0032] In step 130, the original image, the initial edge information of the target object, and the desired edge information of the target object are input into the image generation model, which then outputs a target image. Here, the target image includes the target object with the desired edge information. Continuing with the example above, the image generation model can output a target image containing a "mouth" with edge information consistent with a "closed mouth state."

[0033] In some examples of the embodiments of the present application, the image generation model can be a generator model in a generative adversarial network, and each training sample in the training sample set of the image generation model (or, the generator model) includes an image having a set object and training target edge information for the set object, and the set object includes a target object. Here, the training target edge information can represent edge information edited in the training sample, such as edge information obtained by editing the original edge information through various editing methods. Thus, by utilizing the powerful generalization ability of the generator model, a target image of a target object with desired edge information can achieve a realistic effect.

[0034] It should be noted that in the current related technologies, in order to achieve the generalization ability of the generative adversarial network, it is generally necessary to input random Gaussian noise (for example, input by the dropout layer) into the generator model of the generative adversarial network. However, since the Gaussian noise input is uncontrollable, it can only ensure the generalization and ignore the stability of the generated results, which may, for example, result in unsatisfactory generated images.

[0035] In contrast, in the embodiments of the present application, the generative adversarial network is effectively integrated and improved, using an image with an object (e.g., an RGB image) as input (instead of Gaussian noise), and adding the edge information of the object in the image (i.e., the initial edge information and the target edge information) as a conditional constraint without arbitrary generalization, so that the generated image can achieve a very realistic effect.

[0036] In some embodiments, regarding step 130 described above, the target object having the desired edge information can be determined based on the image generation model using the initial edge information and the desired edge information of the target object. Furthermore, based on the image generation model, the target object having the desired edge information can be used to replace the target object having the initial edge information in the original image to generate the target image. Thus, while the target object in the original image is modified, image information other than the target object in the original image can be retained and fused.

[0037] It should be noted that, regarding the above step 110 , an edge extraction algorithm in the traditional computer vision field may be used to extract the edge information of the object. However, there may be problems in that the extracted edge information is not clear enough and the efficiency is low.

[0038] In light of this, embodiments of the present application propose using a deep learning network to replace traditional computer vision edge detection algorithms to extract initial edge information of at least one object in the original image. In some examples of the embodiments of the present application, a pix2pix network can be used to detect edge information of objects in the image. This achieves significant improvements in both generation quality and efficiency.

[0039] Figure 2 A flowchart of an example of determining desired edge information in an image generation method according to an embodiment of the present application is shown.

[0040] like Figure 2 As shown, in step 210, the initial edge information available for editing in the original image is displayed. In some application scenarios, the user can edit the initial edge information of all objects in the image, or the user can only edit the initial edge information of some objects in the image. In some embodiments, the user can (or can only) edit the edge information of the objects in the corresponding set object set in the original image. Exemplarily, the set object set can be determined based on the object types covered in the training sample set of the image generation model (or the object types that the image generation model can generate). For example, when the images of each training sample in the training sample set do not have the object "ear", the image generation model cannot generate an image with the object "ear". In this way, even if the edge information of the "ear" in the original image is extracted, the user still cannot edit the edge information of the "ear".

[0041] In step 220, an image editing operation is performed on the displayed initial edge information to determine the corresponding target object and the desired edge information of the target object. For example, assuming the original image is a frontal face image, the frontal face image contains information about the facial features (e.g., eyes, nose, and mouth), and has initial edge information corresponding to an open mouth. In this case, the user can use edge information operations to adjust the edge information to that of a closed mouth.

[0042] Figure 3 A flowchart of an example of constructing a generative adversarial network with an image generation model according to an embodiment of the present application is shown.

[0043] like Figure 3 As shown, in step 310, the input of the generative adversarial network is defined as an image (e.g., an RGB image) with an arbitrary object (or set object).

[0044] In step 320, the generator model in the generative adversarial network is constructed using the UNet network as the base network. Figure 4 A schematic diagram of the structure of an example of UNet is shown. Figure 4 As shown, one side of the input end of UNet (the left side of the lowest point of UNet in the figure) continuously encodes and reduces the dimension of the input image blocks, and the other side (the right side of the lowest point of UNet in the figure) continuously decodes and increases the dimension of the data output by the last encoded layer.

[0045] In step 330, a self-attention layer (Attention Map) is configured in the generator model. For example, the last encoding layer of the UNet network is connected to the first self-attention network, and the last decoding layer of the UNet network is connected to the second self-attention network, wherein the first self-attention network includes a first preset number of self-attention layers, and the second self-attention network includes a second preset number of self-attention layers. Here, the self-attention layer is conditionally inputted based on the target edge information of the target object (e.g., the edge information of the edited target object). In this way, the generator model can generate a new image based on the edge information of the edited object without overgeneralization. Specifically, the original image can be encoded based on each encoding layer of the UNet network (e.g., each layer on the left) to obtain a corresponding first encoded image. The desired edge information can also be encoded into the first encoded image based on the first self-attention network to obtain a corresponding second encoded image. The second encoded image is decoded based on each decoding layer of the UNet network (e.g., each layer on the right) and the second self-attention network to obtain a target image of the target object with the desired edge information. Figure 5A structural diagram of an example of a self-attention network is shown. Conditional constraints can be added to the self-attention network to obtain a self-attention map, which can solve the problem of over-generalization of the network.

[0046] In step 340, the loss functions for the generator model in the two training phases are configured. It should be noted that during the training process of a generative adversarial network, the generator model in the generative adversarial network is trained separately in the first phase. After the training of the generator model is completed, the second phase begins, where both the generator model and the discriminator model are trained and updated simultaneously. Various loss functions commonly used in generative adversarial networks can be used here, and are not limited here.

[0047] In some examples of the embodiments of the present application, a simple L1 loss function can be used in the first stage: L1(G)=Σ[||y i -G(x i )||1], in the second stage, the loss function can be L(G)=-E z～pz,y～pdata D(G(z),y), where x represents the real input image, y represents the real expected image, i represents the sum lower bound, G represents the generator model, which creates or generates realistic images from the input image, D(G(z),y) represents the discriminator model, whose input is the image G(z) generated by the generator and the real expected image y, E z～pz,y～pdata Indicates the gap between the generator-generated image and the actual expected image.

[0048] In step 350, a self-attention layer is configured in the discriminator model. For example, as in the generator model, a self-attention layer can be added to the last two convolutional layers. This can enhance the discriminator model's ability to distinguish true from false.

[0049] In step 360, the Hinge loss function is set as the loss function of the discriminator model in the generative adversarial network:

[0050] L D =-E (x,y)～pdata [min(0,-1+D(x,y))]-E z～pz,y～pdata [min(0,-1-D(G(z),y))]

[0051] Among them, E z～pz,y～pdata (m) represents the average loss of m.

[0052] Compared with other types of loss functions, using the Hinge loss function can effectively improve the performance of the generative adversarial network and ensure that the generator model in the generative adversarial network can generate realistic images.

[0053] In the embodiments of the present application, the discriminative model and the generative model are effectively integrated, which not only makes up for the disadvantage of poor generalization ability of the discriminative network but also avoids the disadvantage of the difficult control of the results generated by the generative network. At the same time, the idea of ​​SAGAN (self-attention generative adversarial network) is introduced, which further improves the image restoration effect on the existing technology.

[0054] Figure 6 A flowchart of an example of training an image generation model according to an embodiment of the present application is shown.

[0055] like Figure 6 As shown, in step 610, a training sample set for the image generation model is determined.

[0056] Specifically, each training sample in the training sample set can be an image of a set object (e.g., an arbitrary target object or a target object that matches the set object set). In addition, the training sample also needs to have edge information of the current state (e.g., state 1) of the set object (e.g., object a) and edge information of the corresponding target state (state 2), and the base image can be an image of object a in state 2. The output can be a predicted image (or generated image) of a in state 2 determined by the generator model.

[0057] Regarding the method of extracting edge information of objects in each training sample, it can be extracted through a deep learning network (e.g., a pix2pix network). For example, after the edge information of object a (e.g., edge information in state 1) is extracted through the pix2pix network, the target edge information of object a in other states (e.g., the object state after being edited) (e.g., edge information in state 2 or other states) can be determined by setting object displacement rules or object displacement constraints (e.g., opening and closing of the mouth, rotation of the arm, etc.).

[0058] In addition, the training sample set should also have images of the object in various other states as the base image (or target image) to be used by the discriminator model in the generative adversarial network, that is, to identify whether the image is a generated image or a real base image. For example, a series of images of the target object in different states can be manually collected to construct the original image and the corresponding target image. In some embodiments, at least part of the training samples in the training sample set are determined based on video frame images (e.g., continuous video frames) with a set object. For example, using a video of a person talking continuously, there are open and closed states of the person's mouth in the continuous video frame images, and then the image in the closed state can be used as the original image, and the image in the open state can be used as the target image.

[0059] In step 620, the generator model in the generative adversarial network is trained separately.

[0060] In step 630, when the generator model is iterated to a certain extent, the generator model and the discriminator model in the generative adversarial network are trained simultaneously.

[0061] It should be noted that, since the input of the generative adversarial network is an image with an object (for example, an arbitrary object), the generated image is also an image with an object. If the generator model and the discriminator model are updated from the beginning at the same time, the training difficulty of the generator will be greatly increased, and the generator model may even collapse. In an embodiment of the present application, in order to avoid this situation, the generator model needs to be trained first and the loss function L1 of the first stage is used. In addition, when the generator effect is close to optimal, the generator and the discriminator are trained at the same time. At this time, the generator uses the loss function L(G) of the second stage, and the loss function used by the discriminator model remains unchanged. Through the training method of the embodiment of the present application, it can be guaranteed that the generative adversarial network has higher performance.

[0062] In an embodiment of the present application, an image with a target object is used as input, and the edited target object edge object information image is used as a conditional constraint of the generator network. When the generator network is trained alone, the network can be effectively converged, which not only shortens the training time of the entire adversarial generative network, but also ensures the generation effect when the generator model and the discriminator model are trained at the same time.

[0063] Furthermore, in the testing phase or the use phase of the generator model, it is only necessary to input the image corresponding to the target object and the edited edge information, and finally output the image of the target object with the edited edge information state.

[0064] In an embodiment of the present application, Gaussian noise is no longer used as the input of the generator model, but an image with a target object is used as input, so that the generated image is controllable. Secondly, during the training process, the target object state (for example, different edge information of the object) that is expected to be generated is gradually added through the self-attention layer as a conditional constraint for network training. The generative adversarial network in the embodiment of the present application is based on UNet and self-attention mechanism as the main ideas, combined with the respective characteristics of the discriminant model and the generative model, and under the guidance of edge information as a conditional constraint, not only the training speed is fast but also very realistic edited objects can be generated.

[0065] In some application scenarios of the embodiments of the present application, when a user needs to edit a target object, the user only needs to edit the edge information corresponding to the target object to generate an image of the object in the target state.

[0066] Figure 7A schematic diagram shows the effects of the image generation method according to an embodiment of the present application in some application scenarios.

[0067] like Figure 7 As shown, the embodiment of the present application can be applied in an image editing application (or APP). For example, when a user opens an image editing application, the user can input the original image to be edited, and the application can extract the edge information in the original image. At this time, the user can edit the edge information of the area desired to be edited. Figure 7 In the example, Figure 7 The edge information of the girl's arm in the original image is moved to the edge information at the target position. The generative adversarial network generates an image of the girl with the left arm at the target position (i.e., the girl's left elbow is exposed) based on the original edge information of the arm in the original image, the edge information at the target position, and the original image. Exemplarily, the generative adversarial network can generate a left arm at the target position to replace the left arm in the original image, and fuse or merge the image information of other parts of the original image (for example, except for the left arm) to obtain the target image. Therefore, the shape of the target object can be changed by simple sketching, etc., thereby achieving the purpose of portrait and image editing.

[0068] It should be understood that such image editing applications can be applied to various terminals, such as mobile phones, televisions, and all other terminal devices that generate facial animation videos in two stages, which can improve the system user experience of human-computer interaction.

[0069] Figure 8 A structural block diagram of an example of an image generating device according to an embodiment of the present application is shown.

[0070] like Figure 8 As shown, the image generating apparatus 800 includes an initial edge information extracting unit 810 , an edited edge information acquiring unit 820 , a target image generating unit 830 , and an object edge displaying unit 840 .

[0071] The initial edge information extraction unit 810 is configured to extract the initial edge information of at least one object in the original image. Figure 1 The operation described in step 110.

[0072] The editing edge information acquisition unit 820 is configured to acquire an image editing operation and determine the target object to be edited and the desired edge information of the target object based on the image editing operation. Figure 1 The operation described in step 120 in FIG.

[0073] The target image generation unit 830 is configured to input the original image, the initial edge information of the target object, and the desired edge information of the target object into the image generation model, and obtain the target image output by the image generation model, wherein the target image includes the target object with the desired edge information. For details of the target image generation unit 830, please refer to the reference above. Figure 1 The operation described in step 130 in FIG.

[0074] In some examples of the embodiments of the present application, the object edge display unit 840 is configured to display initial edge information available for editing. Accordingly, the edit edge information acquisition unit 820 is configured to acquire an image editing operation for the displayed initial edge information to determine the corresponding target object and the desired edge information of the target object.

[0075] It should be noted that the image generating device 800 may include only one or more of the above units. For example, in some examples, the object edge display unit 840 may not be provided in the image generating device 800.

[0076] In some embodiments, the initial edge information extraction unit 810 is configured to extract initial edge information of at least one object in the original image based on a deep learning network.

[0077] In some embodiments, at least part of the training samples in the training sample set are determined based on video frame images having a set object.

[0078] In some embodiments, the edge information of the set object in the training sample and the edge information of the training target meet the preset constraint conditions of the object displacement.

[0079] In some embodiments, the image generation model includes a UNet network, a first self-attention network, and a second self-attention network, the last encoding layer of the UNet network is connected to the first self-attention network, and the last decoding layer of the UNet network is connected to the second self-attention network, wherein the first self-attention network includes a first preset number of self-attention layers, the second self-attention network includes a second preset number of self-attention layers, and the self-attention layers take the edge information of the target object as conditional input.

[0080] In some embodiments, the loss function used by the discriminator model in the generative adversarial network is a folding loss function.

[0081] As above Figures 1 to 8, describes embodiments of the image generation method and apparatus according to embodiments of the present application. The details mentioned in the above description of the method embodiment also apply to the embodiments of the apparatus of the present application. The above image generation apparatus can be implemented using hardware, software, or a combination of hardware and software.

[0082] Figure 9 FIG. 1 shows a hardware structure diagram of an example of an electronic device 900 for generating an image according to an embodiment of the present application. Figure 9 As shown, the electronic device 900 may include at least one processor 910, a memory (e.g., a non-volatile memory) 920, a storage 930, and a communication interface 940, and the at least one processor 910, the memory 920, the storage 930, and the communication interface 940 are connected together via a bus 960. The at least one processor 910 executes at least one computer-readable instruction stored or encoded in the memory (i.e., the above-mentioned element implemented in the form of software).

[0083] In one embodiment, computer-executable instructions are stored in a memory, which, when executed, cause at least one processor 910 to: extract initial edge information of at least one object in an original image; obtain an image editing operation, and determine a target object to be edited and desired edge information of the target object based on the image editing operation; input the original image, the initial edge information of the target object, and the desired edge information of the target object into an image generation model to obtain a target image output by the image generation model, wherein the target image includes the target object having the desired edge information.

[0084] It should be understood that the computer executable instructions stored in the memory 920, when executed, cause at least one processor 910 to perform the above combined operations in various embodiments of the present application. Figure 1-7 Describes the various operations and functions.

[0085] In this specification, the electronic device 900 may include but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile electronic devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable electronic devices, consumer electronic devices, etc.

[0086] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in software form), which, when executed by a machine, causes the machine to perform the above-mentioned combination of various embodiments of the present application. Figure 1-7Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes for implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can be enabled to read and execute the instructions stored in the readable storage medium.

[0087] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of the present invention.

[0088] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0089] Those skilled in the art will appreciate that various variations and modifications may be made to the above embodiments without departing from the essence of the invention. Therefore, the scope of protection of the present invention shall be defined by the appended claims.

[0090] It should be noted that not all steps and units in the above processes and system structure diagrams are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical entity, or some units may be implemented by multiple physical entities, or may be implemented by certain components in multiple independent devices.

[0091] In the above embodiments, hardware unit or module can be realized by mechanical means or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. Hardware unit or processor can also include programmable logic or circuit (such as general purpose processor or other programmable processor), can be temporarily set up to complete the corresponding operation by software. Concrete implementation (mechanical means or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0092] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0093] The foregoing description of the present invention is provided to enable any person skilled in the art to implement or use the present invention. Various modifications to the present invention will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present invention. Therefore, the present invention is not limited to the examples and designs described herein, but is intended to be consistent with the widest scope of the principles and novel features of the present invention.

Claims

1. An image generation method, characterized in that: include: extracting initial edge information of at least one object in the original image; Acquiring an image editing operation, and determining a target object to be edited and desired edge information of the target object based on the image editing operation; The original image and the desired edge information of the target object are input into an image generation model to obtain a target image output by the image generation model, wherein the target image includes the target object with the desired edge information; the image generation model is a generator model in a generative adversarial network, and the generator model is based on a UNet network; the last encoding layer of the UNet network is connected to a first self-attention network, and the last decoding layer of the UNet network is connected to a second self-attention network, wherein the first self-attention network includes a first preset number of self-attention layers, and the second self-attention network includes a second preset number of self-attention layers; the self-attention layer uses the edge information of the edited target object as a conditional input, so that the generator model can generate a new image with the edge information of the edited object as the target.

2. The image generation method according to claim 1, wherein: The acquiring of the image editing operation and determining the target object to be edited and the expected edge information of the target object based on the image editing operation include: Displays initial edge information for editing; An image editing operation is obtained for the displayed initial edge information to determine a corresponding target object and desired edge information of the target object.

3. The image generation method according to claim 1, wherein: Extracting initial edge information of at least one object in the original image, specifically including: Based on a deep learning network, the initial edge information of at least one object in the original image is extracted.

4. The image generation method according to claim 1, wherein: Each training sample in the training sample set of the image generation model includes an image having a set object and training target edge information for the set object, the set object includes the target object, wherein the edge information of the set object in the training sample and the training target edge information meet the preset object displacement constraint conditions.

5. The image generation method according to claim 1 or 4, wherein: At least part of the training samples in the training sample set of the image generation model is determined based on video frame images having a set object.

6. The image generation method according to claim 1, wherein: Inputting the original image and the desired edge information of the target object into an image generation model to obtain a target image of the target object having the desired edge information output by the image generation model includes: Encoding the original image based on each encoding layer in the UNet network to obtain a corresponding first encoded image; encoding the desired edge information into the first encoded image based on the first self-attention network to obtain a corresponding second encoded image; Based on the respective decoding layers in the UNet network and the second self-attention network, the second encoded image is decoded to obtain a target image of the target object having the desired edge information.

7. The image generation method according to claim 1, wherein: Inputting the original image and the desired edge information of the target object into an image generation model to obtain a target image of the target object having the desired edge information output by the image generation model includes: Based on the image generation model, Determining a target object having the expected edge information by using the initial edge information of the target object and the expected edge information, and The target object having the initial edge information in the original image is replaced by the target object having the desired edge information to generate the target image.

8. An image generating device, comprising: an initial edge information extraction unit, configured to extract initial edge information of at least one object in an original image; an editing edge information acquiring unit configured to acquire an image editing operation, and determine a target object to be edited and desired edge information of the target object based on the image editing operation; A target image generation unit is configured to input the original image and the desired edge information of the target object into an image generation model to obtain a target image output by the image generation model, wherein the target image includes the target object with the desired edge information; the image generation model is a generator model in a generative adversarial network, and the generator model is based on a UNet network; the last encoding layer of the UNet network is connected to a first self-attention network, and the last decoding layer of the UNet network is connected to a second self-attention network, wherein the first self-attention network includes a first preset number of self-attention layers, and the second self-attention network includes a second preset number of self-attention layers; the self-attention layer uses the edge information of the edited target object as a conditional input, so that the generator model can generate a new image with the edge information of the edited object as the target.

9. An electronic device comprising: at least one processor; as well as A memory storing instructions, which, when executed by the at least one processor, causes the at least one processor to perform the method according to any one of claims 1 to 7.

10. A machine-readable storage medium storing executable instructions, which, when executed, cause the machine to perform the method according to any one of claims 1 to 7.

Images