Method and device for constructing image editing model
By simultaneously modeling the foreground, mask and background of the image in the generative adversarial network, the problem of missing background in the image editing model is solved, the model construction steps are simplified, resource consumption is reduced, and editing performance is improved.
Patent Information
- Application Number
- CN202010576505.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-06-22
Smart Images

Figure CN111861955B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet technology, and in particular to a method, device, electronic device and computer-readable storage medium for constructing an image editing model in the field of image processing and deep learning technology. Background Art
[0002] There are usually two existing solutions for building image editing models: one is to generate a new image by modeling the image as a whole without distinguishing the foreground and background of the image; the other is to use different models to model the foreground and background of the image separately, and fuse the generated foreground and background to generate a new image. For the first solution, since the foreground and background of the image are not distinguished, the constructed model cannot fill the missing background in the generated image due to the reduction of the foreground of the input image; for the second solution, since multiple models are needed to model the foreground and background of the image separately, the model construction steps are more cumbersome and the system resource consumption is high. Summary of the invention
[0003] The technical solution adopted by the present application to solve the technical problem is to provide a method for constructing an image editing model, including: obtaining a training sample, wherein the training sample includes a first image and its corresponding second image; constructing a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch; training the generative adversarial network according to the first image and its corresponding second image, and using the generator in the generative adversarial network obtained by training as an image editing model.
[0004] The technical solution adopted by the present application to solve the technical problem is to provide a device for constructing an image editing model, including: an acquisition unit, used to acquire training samples, wherein the training samples include a first image and its corresponding second image; a construction unit, used to construct a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch; a training unit, used to train the generative adversarial network according to the first image and its corresponding second image, and use the generator in the generative adversarial network obtained by training as the image editing model.
[0005] An electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.
[0006] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.
[0007] A computer program product comprises a computer program, wherein the computer program implements the above method when executed by a processor.
[0008] One embodiment of the above application has the following advantages or beneficial effects: the image editing model constructed by the present application can fill in the missing background of the generated image, and can simplify the model construction steps and reduce the resource consumption required for model construction. Because the technical means of simultaneously modeling the foreground image, mask image and background image when constructing the generative adversarial network is adopted, the technical problems caused by not distinguishing the foreground and background of the image in the prior art and requiring multiple models to model the foreground and background separately are overcome, and the technical effects such as improving the editing performance of the image editing model are achieved.
[0009] Other effects of the above optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0011] Figure 1 is a schematic diagram according to the first embodiment of the present application;
[0012] Figure 2 is a schematic diagram according to a second embodiment of the present application;
[0013] Figure 3 is a schematic diagram according to a third embodiment of the present application;
[0014] Figure 4 It is a block diagram of an electronic device used to implement the method of constructing an image editing model in an embodiment of the present application. DETAILED DESCRIPTION
[0015] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0016] Figure 1 is a schematic diagram according to the first embodiment of the present application. Figure 1 As shown in , the method for constructing an image editing model in this embodiment may specifically include the following steps:
[0017] S101, obtaining a training sample, where the training sample includes a first image and a corresponding second image;
[0018] S102, constructing a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch;
[0019] S103: training the generative adversarial network according to the first image and its corresponding second image, and using the generator in the generative adversarial network obtained through training as an image editing model.
[0020] The method for constructing an image editing model in this embodiment adopts a method of simultaneously modeling the foreground image, the mask image and the background image when constructing a generative adversarial network, so that the trained image editing model can fill in the missing background of the generated image, thereby avoiding the problem of missing background of the generated image due to the reduction of the foreground of the input image.
[0021] The training samples acquired in this embodiment include a first image and a corresponding second image, wherein the second image can be considered as an image with a complete background obtained when editing the first image.
[0022] For example, if the first image is a female image, the second image may be a male image. This is because when editing a female image into a male image, the female's long hair needs to be converted into the male's short hair, which results in the missing hair foreground in the female image needing to be turned into the background in the male image, but this part of the background is missing in the female image, so it is necessary to fill in the missing background when editing the female image to obtain the male image.
[0023] If the first image is a male image, the second image can be a baby image. This is because when editing a male image into a baby image, the male's short hair needs to be converted into a baby's hairless hair, which results in the missing hair foreground in the male image needing to be turned into the background in the baby image, but this part of the background is missing in the male image, so it is necessary to fill the missing background when editing the male image to obtain the baby image.
[0024] When acquiring training samples, the present embodiment can directly set the first image and its corresponding second image according to the acquired image. When setting the correspondence between images, the present embodiment can manually set the first image and its corresponding second image according to actual needs by the user; or automatically set the first image and its corresponding second image according to a preset content correspondence after performing content recognition on each image.
[0025] In addition, when acquiring training samples, this embodiment can also adopt the following method: acquiring a foreground image and a background image, wherein the foreground image is an image of a person or an object; respectively fusing two foreground images having a corresponding relationship with the same background image, and using the fusion result as a first image and its corresponding second image. In this embodiment, the user can manually select two foreground images having a corresponding relationship; or the user can automatically select two foreground images having a corresponding relationship after performing content recognition on the image.
[0026] That is to say, this embodiment adopts the method of fusing two foreground images with the same background image to obtain training samples, which can ensure that each first image and its corresponding second image have the same background, so that the image editing model can more naturally fill in the missing background in the input image, thereby improving the accuracy of the image editing model in filling the background.
[0027] After obtaining the training samples, this embodiment performs an operation of constructing a generative adversarial network including a generator and a discriminator, thereby obtaining an image editing model capable of filling in the missing background in the generated image based on the generator in the trained generative adversarial network. The generative adversarial network constructed in this embodiment belongs to a deep learning neural network.
[0028] In the adversarial generative network constructed in this embodiment, the generator is responsible for generating generated samples that are as similar as possible to real samples, and the discriminator is responsible for distinguishing between real samples and generated samples as much as possible. The generative adversarial network is trained by adversarial game between the generator and the discriminator, so that the authenticity of the generated samples output by the generator is as high as possible, so that the discriminator cannot distinguish whether the output obtained by the generative model is a generated sample or a real sample.
[0029] Specifically, the generator in the generative adversarial network constructed in this embodiment includes three image generation branches, namely, a foreground image generation branch, a mask image generation branch, and a background image generation branch, each of which generates a foreground image, a mask image, and a background image according to the acquired image features. Among them, the network structure of the three image generation branches included in the generator is a deep neural network, which can be a convolutional neural network, a recurrent neural network, a recursive neural network, etc.
[0030] That is to say, this embodiment can avoid using multiple models for separate modeling by simultaneously modeling the foreground image, mask image, and background image in the generator in the generative adversarial network, thereby simplifying the steps of building the image editing model and reducing the resource consumption required to build the image editing model.
[0031] After constructing a generative adversarial network including a generator and a discriminator, this embodiment trains the generative adversarial network based on the first image and its corresponding second image in the training sample, so that the generator in the trained generative adversarial network is used as an image editing model, which can fill in the missing background in the generated image.
[0032] This embodiment uses an alternating training method to train a generative adversarial network constructed by a generator and a discriminator. When the entire generative adversarial network converges, the training of the generative adversarial network is considered to be completed, and the generator in the trained generative adversarial network is used as an image editing model.
[0033] Specifically, when training a generative adversarial network, this embodiment can adopt the following method: use the second image in the training sample as a real sample; after inputting the first image into the generator, first obtain the image feature of the first image, which is the deep semantic information of the image represented by a vector, and then input the image feature into the foreground image generation branch, the mask image generation branch, and the background image generation branch respectively, and finally fuse the generated foreground image, mask image, and background image to obtain a generated sample; use the generated sample and its corresponding real sample as input to the discriminator; alternately train the generator and the discriminator until the generative adversarial network converges.
[0034] In this embodiment, when alternately training the generator and the discriminator until the generative adversarial network converges, the damage functions of the corresponding generator and the discriminator are obtained based on the output results of the discriminant model, and then the parameters in the network structure of the generator and the discriminator are adjusted according to the damage functions of the generator and the discriminator until the adversarial network converges.
[0035] In this embodiment, when adjusting the parameters of the network structure of the generator and the discriminator according to the loss function, the training objectives of the generator and the discriminator are respectively to minimize the loss function. Optionally, in a specific implementation process of this embodiment, if the loss function obtained within a preset number of times converges, the loss function is considered to be minimized; it can also be considered that the loss function is minimized when the loss function obtained converges to a preset value; it can also be considered that the loss function is minimized when the number of training times exceeds a preset number of times.
[0036] When the generator in this embodiment fuses the foreground image, the mask image and the background image, it specifically uses the content of the foreground image for the area with a pixel value of 1 in the mask image, and uses the content of the background image for the area with a pixel value of 0 in the mask image, thereby obtaining a fused image.
[0037] It can be understood that since the generator in this embodiment includes three image generation branches, and each image branch is a neural network, the loss function of the generator in this embodiment can be composed of the loss functions corresponding to the three image generation branches. When the loss functions of the three image generation branches are minimized, it can be considered that the damage function of the generator is minimized. Therefore, this embodiment can improve the adjustment accuracy of the parameters in the network structure of each branch by setting the loss functions of the three image generation branches separately, so that the generator can generate generated samples that are more similar to real samples.
[0038] Among them, the loss function corresponding to the mask image generation branch in this embodiment is used to constrain the generated mask image to be consistent with the foreground and background segmentation results of the real sample; the damage function corresponding to the foreground image generation branch is used to constrain the generated foreground image to be as close to the foreground image of the real sample as possible, that is, the generated sample is as close to the real sample as possible; the loss function corresponding to the background image generation branch is used to constrain the generated background to be consistent with the background of the real sample.
[0039] The image editing model obtained by the above method of this embodiment, by simultaneously modeling the foreground, background and mask of the image in one model, can, on the one hand, solve the problem of missing background of the generated image due to the reduction of the foreground of the input image, thereby improving the editing performance of the image editing model; on the other hand, it avoids the problems of cumbersome steps and high system resource consumption caused by using different models to model the foreground, background and mask respectively, simplifies the steps of building the image editing model, and achieves the purpose of building the image editing model with lower system resource consumption.
[0040] Figure 2 It is a schematic diagram according to the second embodiment of the present application. Figure 2 The editing process of the image editing model in this embodiment is shown. Figure 2 As shown in , after the input image is input into the image editing model, the image features of the input image are first extracted, and then the image features are respectively input into the foreground image generation branch, the mask image generation branch and the background image generation branch, and finally the generated foreground image, mask image and background image are fused, and the fusion result is used as the final generated image.
[0041] Figure 3 is a schematic diagram according to the third embodiment of the present application. Figure 3 As shown in , the device for constructing an image editing model of this embodiment includes:
[0042] An acquisition unit 301 is used to acquire a training sample, where the training sample includes a first image and a corresponding second image;
[0043] A construction unit 302 is used to construct a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch;
[0044] The training unit 303 is used to train the generative adversarial network according to the first image and its corresponding second image, and use the generator in the generative adversarial network obtained through training as an image editing model.
[0045] The training samples acquired by the acquisition unit 301 include a first image and a corresponding second image, wherein the second image can be considered as an image with a complete background obtained when editing the first image.
[0046] When acquiring training samples, the acquisition unit 301 can directly set the first image and its corresponding second image according to the acquired image. When setting the correspondence between images, the acquisition unit 301 can manually set the first image and its corresponding second image according to actual needs by the user; or automatically set the first image and its corresponding second image according to a preset content correspondence after performing content recognition on each image.
[0047] In addition, when acquiring training samples, the acquisition unit 301 may also adopt the following method: acquiring a foreground image and a background image, wherein the foreground image is an image of a person or an object; respectively fusing two foreground images having a corresponding relationship with the same background image, and using the fusion result as a first image and its corresponding second image. The acquisition unit 301 may manually select two foreground images having a corresponding relationship by a user; or automatically select two foreground images having a corresponding relationship after performing content recognition on the images.
[0048] In this embodiment, after the acquisition unit 301 acquires the training sample, the construction unit 302 performs the operation of constructing a generative adversarial network including a generator and a discriminator, thereby obtaining an image editing model that can fill in the missing background in the generated image based on the generator in the trained generative adversarial network.
[0049] In the adversarial generative network constructed by the construction unit 302, the generator is responsible for generating generated samples that are as similar as possible to real samples, and the discriminator is responsible for distinguishing between real samples and generated samples as much as possible. The generative adversarial network is trained by adversarial game between the generator and the discriminator, so that the authenticity of the generated samples output by the generator is as high as possible, so that the discriminator cannot distinguish whether the output obtained by the generative model is a generated sample or a real sample.
[0050] Specifically, the generator in the generative adversarial network constructed by the construction unit 302 includes three image generation branches, namely a foreground image generation branch, a mask image generation branch, and a background image generation branch, and each branch generates a foreground image, a mask image, and a background image according to the acquired image features. Among them, the network structure of the three image generation branches included in the generator is a deep neural network, which can be a convolutional neural network, a recurrent neural network, a recursive neural network, etc.
[0051] After the construction unit 302 constructs a generative adversarial network including a generator and a discriminator, the training unit 303 trains the generative adversarial network based on the first image and its corresponding second image in the training sample, and uses the generator in the trained generative adversarial network as an image editing model, which can fill in the missing background in the generated image.
[0052] The training unit 303 trains the generative adversarial network constructed by the generator and the discriminator in an alternating training manner. When the entire generative adversarial network converges, the training of the generative adversarial network is considered to be completed, and the generator in the trained generative adversarial network is used as an image editing model.
[0053] Specifically, when training the generative adversarial network, the training unit 303 can adopt the following method: take the second image in the training sample as the real sample; after inputting the first image into the generator, first obtain the image features of the first image and then input the image features into the foreground image generation branch, the mask image generation branch and the background image generation branch respectively, and finally fuse the generated foreground image, mask image and background image to obtain the generated sample; take the generated sample and its corresponding real sample as the input of the discriminator; alternately train the generator and the discriminator until the generative adversarial network converges.
[0054] The training unit 303 alternately trains the generator and the discriminator until the generative adversarial network converges. Specifically, the damage function of the corresponding generator and the discriminator is obtained according to the output result of the discriminant model, and then the parameters in the network structure of the generator and the discriminator are adjusted according to the damage function of the generator and the discriminator until the adversarial network converges.
[0055] When the training unit 303 adjusts the parameters of the network structure of the generator and the discriminator according to the loss function, the training objectives of the generator and the discriminator are respectively to minimize the loss function. Optionally, in a specific implementation process of the training unit 303, if the loss function obtained converges within a preset number of times, the loss function is considered to be minimized; it can also be considered that the loss function is minimized when the loss function obtained converges to a preset value; it can also be considered that the loss function is minimized when the number of training times exceeds a preset number of times.
[0056] It can be understood that since the generator constructed by the construction unit 302 includes three image generation branches, and each image branch is a neural network, the loss function of the generator in the training unit 303 can be composed of the loss functions corresponding to the three image generation branches. When the loss functions of the three image generation branches are minimized, it can be considered that the damage function of the generator is minimized.
[0057] Among them, the loss function corresponding to the mask image generation branch in the training unit 303 is used to constrain the generated mask image to be consistent with the foreground and background segmentation results of the real sample; the damage function corresponding to the foreground image generation branch is used to constrain the generated foreground image to be as close to the foreground image of the real sample as possible, that is, the generated sample is as close to the real sample as possible; the loss function corresponding to the background image generation branch is used to constrain the generated background to be consistent with the background of the real sample.
[0058] According to an embodiment of the present application, the present application also provides an electronic device, a computer-readable storage medium, and a computer program product.
[0059] like Figure 4 , is a block diagram of an electronic device for constructing a method for an image editing model according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0060] like Figure 4 As shown, the electronic device includes: one or more processors 401, a memory 402, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 401 is taken as an example.
[0061] The memory 402 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the method for constructing an image editing model provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to execute the method for constructing an image editing model provided in the present application.
[0062] The memory 402 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the method for constructing an image editing model in the embodiment of the present application (for example, the attached Figure 3 The processor 401 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 402, that is, the method of constructing the image editing model in the above method embodiment is implemented.
[0063] The memory 402 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory 402 may optionally include a memory remotely arranged relative to the processor 401, and these remote memories may be connected to the electronic device of the method for constructing the image editing model via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0064] The electronic device of the method for constructing an image editing model may further include: an input device 403 and an output device 404. The processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0065] The input device 403 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the method for constructing the image editing model, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 404 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0066] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0067] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0068] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0069] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0070] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0071] According to the technical solution of the embodiment of the present application, by simultaneously modeling the foreground, background and mask of the image in one model, on the one hand, it can solve the problem of missing background of the generated image due to the reduction of the foreground of the input image, thereby improving the editing performance of the image editing model; on the other hand, it avoids the problems of cumbersome steps and high system resource consumption caused by using different models to model the foreground, background and mask respectively, simplifies the steps of building the image editing model, and achieves the purpose of building the image editing model with lower consumption of system resources.
[0072] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0073] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A method for constructing an image editing model, comprising: Acquire a training sample, where the training sample includes a first image and a corresponding second image, and the age attribute or gender attribute of a person in the first image and the corresponding second image is different; Constructing a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch, and the background image generation branch, the mask image generation branch, and the foreground image generation branch are used to generate a background image, a mask image, and a foreground image respectively according to the same image features, and the image features are deep semantic information of the image represented by a vector extracted by the generator from the input image; The generative adversarial network is trained according to the first image and its corresponding second image, and the generator in the generative adversarial network obtained by training is used as an image editing model. The image editing model is used to change the age attribute or gender attribute of a person in the input image, and to fill in the missing background in the generated image due to the change in the age attribute or gender attribute.
2. The method according to claim 1, wherein: The obtaining of training samples comprises: Get the foreground image and the background image; Two foreground images having a corresponding relationship are fused with the same background image respectively, and the fusion result is used as the first image and the corresponding second image in the training sample.
3. The method according to claim 1, wherein: The network structure of the background image generation branch, the mask image generation branch and the foreground image generation branch is a deep neural network.
4. The method according to claim 1, wherein: The training of the generative adversarial network according to the first image and its corresponding second image comprises: Taking the second image in the training sample as a real sample; After the first image is input into the generator, image features are extracted and respectively input into the foreground image generation branch, the mask image generation branch and the background image generation branch, and the images generated by each branch are fused to obtain a generated sample; The generated samples and their corresponding real samples are used as inputs to the discriminator; The generator and the discriminator are trained alternately until the generative adversarial network converges.
5. The method according to claim 4, wherein: The alternately training the generator and the discriminator until the generative adversarial network converges includes: For the three image generation branches contained in the generator, construct the loss function corresponding to each image generation branch; According to the loss function corresponding to each image generation branch, the parameters in the network structure of each image generation branch are adjusted respectively until the generative adversarial network converges.
6. A device for constructing an image editing model, comprising: An acquisition unit, configured to acquire a training sample, wherein the training sample includes a first image and a corresponding second image, and the age attribute or gender attribute of a person in the first image and the corresponding second image is different; A construction unit, configured to construct a generative adversarial network including a generator and a discriminator, wherein the generator includes a background image generation branch, a mask image generation branch, and a foreground image generation branch, wherein the background image generation branch, the mask image generation branch, and the foreground image generation branch are configured to respectively generate a background image, a mask image, and a foreground image according to the same image features, wherein the image features are deep semantic information of the image represented by a vector extracted by the generator from the input image; A training unit is used to train the generative adversarial network according to the first image and its corresponding second image, and use the generator in the generative adversarial network obtained by training as an image editing model, wherein the image editing model is used to change the age attribute or gender attribute of a person in the input image, and fill in the missing background in the generated image due to the change of the age attribute or gender attribute.
7. The device according to claim 6, wherein: When acquiring the training sample, the acquisition unit specifically performs: Get the foreground image and the background image; Two foreground images having a corresponding relationship are fused with the same background image respectively, and the fusion result is used as the first image and the corresponding second image in the training sample.
8. The device according to claim 6, wherein: The network structure of the background image generation branch, the mask image generation branch and the foreground image generation branch constructed by the construction unit is a deep neural network.
9. The device according to claim 6, wherein: When the training unit trains the generative adversarial network according to the first image and the second image corresponding thereto, the training unit specifically performs: Taking the second image in the training sample as a real sample; After the first image is input into the generator, image features are extracted and respectively input into the foreground image generation branch, the mask image generation branch and the background image generation branch, and the images generated by each branch are fused to obtain a generated sample; The generated samples and their corresponding real samples are used as inputs to the discriminator; The generator and the discriminator are trained alternately until the generative adversarial network converges.
10. The device according to claim 9, wherein: The training unit trains the generator and the discriminator alternately until the generative adversarial network converges, specifically performing: For the three image generation branches contained in the generator, construct the loss function corresponding to each image generation branch; According to the loss function corresponding to each image generation branch, the parameters in the network structure of each image generation branch are adjusted respectively until the generative adversarial network converges.
11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.
13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Method and apparatus for analyzing medical images
CN107680088A
Facial image conversion method based on cycle generative adversarial network
CN108182657A
A face image restoration method based on a generation antagonism network
CN109377448A
Image generation method, neural network training method and device and electronic equipment
CN109727264A
Video foreground detection method based on full convolutional network and conditional adversarial network
CN110580472A
