Image generation model training method and device, image generation method and device
By adding network modules to the generative adversarial network, combining the generation module and the discriminative network training, the problems of incomplete pictures and insufficient clarity of traditional image generation networks are solved, and a higher quality image generation effect is achieved, which is suitable for multiple Image processing tasks.
Patent Information
- Application Number
- CN202011415838.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-12-07
AI Technical Summary
The traditional generative adversarial network generates incomplete images during image conversion and translation, with many flaws in the local area, and the clarity and detail texture are not high enough, especially for complex backgrounds and uncommon face input, and may even be completely The conversion failed.
The network module is added to the generation network part of the generative adversarial network. By combining the generation module and the network module, the fineness of the generated images is improved. The joint training of the generation module and the discriminative network is used to ensure that the generated images are in completeness, clarity and The richness of details has been significantly improved.
It significantly improves the completeness, clarity and detail richness of the generated images, making the generated images clearer, real and natural, suitable for multiple tasks such as image translation and style conversion, and has strong application value.
Smart Images

Figure CN112529154B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, specifically to the field of image processing, augmented reality, and deep learning technology, and in particular to an image generation model training method and device, an image generation method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0002] The image conversion and translation algorithm network based on the traditional generative adversarial network is generally an encoder and decoder structure. The network directly sends the image generated by the decoder part to the discriminant network for discrimination. The generated image effect is not very good. The generated image is often incomplete, and there are often many defects in the local area. In addition, the generated image clarity is not very high, and the texture of the details is not very clear. Especially for some complex backgrounds and some uncommon face inputs, the generated image effect is often very poor, or even completely fails to convert. Summary of the invention
[0003] Provided are an image generation model training method and device, an image generation method and device, an electronic device, a computer-readable storage medium, and a computer program product.
[0004] According to a first aspect, a method for training an image generation model is provided, the method comprising: obtaining a preset sample set, wherein the sample set comprises at least one sample, and the sample comprises a first domain image and a second domain image; obtaining a pre-established generative adversarial network, wherein the generative network of the generative adversarial network comprises a generation module and a network module, and the network module is used to improve the fineness of the image output by the generation module; performing the following training steps: selecting a sample from the sample set; inputting the first domain image of the sample into the generation module to obtain a pseudo image output by the generation module and a reference image output by the network module; inputting the second domain image of the sample, the pseudo image and the reference image together into the discriminative network of the generative adversarial network; in response to determining that the generative adversarial network meets the training completion condition, using the generative adversarial network as an image generation model.
[0005] According to a second aspect, an image generation model training device is provided, the device comprising: a sample acquisition unit, configured to acquire a preset sample set, wherein the sample set comprises at least one sample, and the sample comprises a first domain image and a second domain image; a network acquisition unit, configured to acquire a pre-established generative adversarial network, wherein the generative network of the generative adversarial network comprises a generation module and a network module, and the network module is used to improve the fineness of the image output by the generation module; a selection unit, configured to select a sample from the sample set; an acquisition unit, configured to input the first domain image of the sample into the generation module, obtain a pseudo image output by the generation module and a reference image output by the network module; an input unit, configured to input the second domain image of the sample, the pseudo image and the reference image into a discriminative network of the generative adversarial network together; a model generation unit, configured to use the generative adversarial network as an image generation model in response to determining that the generative adversarial network meets a training completion condition.
[0006] According to a third aspect, an image generation method is provided, the method comprising: acquiring an image to be processed; inputting the image to be processed into an image generation model generated by the method described in any implementation manner of the first aspect, and outputting the generated image.
[0007] According to a fourth aspect, an image generating device is provided, which includes: an image acquiring unit, configured to acquire an image to be processed; an image generating unit, configured to input the image to be processed into an image generating model generated by the method described in any implementation manner of the first aspect, and output the generated image.
[0008] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.
[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.
[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.
[0011] The technology provided by the embodiment of the present application adds a network module to the generative network part of the generative adversarial network, which improves the fineness of the picture, not only increases the completeness of the picture, but also greatly improves the clarity and detail richness of the picture, making the generated image clearer, more realistic, natural, richer, and with better quality and effect. This technology can be widely used in multiple tasks such as image translation and style transfer, and has strong application value.
[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0014] Figure 1 is a flow chart of an embodiment of the image generation model training method according to the present application;
[0015] Figure 2 It is a structural schematic diagram of the image generation model in this application;
[0016] Figure 3 is a flow chart of an embodiment of an image generation method according to the present application;
[0017] Figure 4 is a structural schematic diagram of an embodiment of an image generation model training device according to the present application;
[0018] Figure 5 is a schematic structural diagram of an embodiment of an image generating device according to the present application;
[0019] Figure 6 It is a block diagram of an electronic device used to implement the image generation model training method or image generation method of an embodiment of the present application. DETAILED DESCRIPTION
[0020] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0021] Figure 1 A process 100 of an embodiment of the image generation model training method according to the present application is shown. The above-mentioned image generation model training method comprises the following steps:
[0022] Step 101: Obtain a preset sample set.
[0023] In this embodiment, the execution subject of the image generation model training method can obtain the sample set in a variety of ways. For example, the execution subject can obtain the existing sample set stored in the database server through a wired connection or a wireless connection. For another example, the user can collect samples through a terminal (such as a terminal connected to the database server). In this way, the execution subject can receive the samples collected by the terminal and store these samples locally, thereby generating a sample set.
[0024] Here, the sample set may include at least one sample. The sample may include a first domain image and a second domain image. Domain refers to the category of the image, for example, male, female, oil painting, photo, cartoon, etc. The sample may include a first domain photo image and a cartoon image converted into a second domain. The image types of the first domain and the second domain are not limited here and may be any combination. For the specific implementation process, please refer to the sample selection step of step 203.
[0025] Step 102: Obtain a pre-established generative adversarial network.
[0026] In this embodiment, the network structure of the generative adversarial network is as follows: Figure 2 As shown in Figure 1, the Generative Adversarial Nets (GAN) includes a generating network G and a discriminating network D. The generating network G of the generative adversarial network includes a generating module g1 and a network module w.
[0027] The generation module consists of an encoder, a converter, and a decoder. The encoder uses a convolutional neural network to extract features from the input image. For example, the image is compressed into 256 64*64 feature vectors. The converter converts the feature vector of the image in the DA domain into a feature vector in the DB domain by combining the dissimilar features of the image. For example, a 6-layer Reset module can be used, each of which is a neural network layer consisting of two convolutional layers, so as to achieve the goal of retaining the original image features during conversion. The decoder uses a deconvolution layer to restore the low-level features from the feature vector, and finally obtains the generated image.
[0028] In this embodiment, the generation network is used to convert the first domain image into the second domain image, and the network module is used to improve the fineness of the image output by the generation module, that is, the network module can improve the fineness of the second domain image. In this embodiment, the fineness includes not only the completeness but also the detail clarity of the image.
[0029] The discriminant network takes an image as input and tries to predict whether it is the original image or the output image of the generative network. The discriminant network itself is a convolutional network, which needs to extract features from the image and then determine whether the extracted features belong to a specific category by adding a convolutional layer that produces a one-dimensional output. In this embodiment, the discriminant network is also used to determine whether the input image is the image output by the generative network.
[0030] It should be noted that the above-mentioned generating network can be a convolutional neural network for image processing (for example, various convolutional neural network structures including convolutional layers, pooling layers, depooling layers, and deconvolution layers, which can be downsampled and upsampled in sequence); the above-mentioned discriminant network can be a convolutional neural network (for example, various convolutional neural network structures including fully connected layers, wherein the fully connected layers can realize classification functions). In addition, the above-mentioned discriminant network can also be other model structures that can be used to realize classification functions, such as support vector machines (SVM).
[0031] Step 103: Select samples from the sample set.
[0032] In this embodiment, the execution subject may select samples from the sample set obtained in step 101, and perform the training steps of steps 103 to 106. Among them, the method and number of sample selection are not limited in this application. For example, at least one sample may be randomly selected, or a sample with better clarity (i.e., higher pixels) may be selected therefrom. Each sample may be a pair of images. A pair of images includes a first domain image and a second domain image. The first domain image and the second domain image may be selected according to actual needs. For example, if the image gender needs to be converted from male to female, a male image is selected as the first domain image, and a female image is selected as the second domain image. Similarly, if the image gender needs to be converted from female to male, a female image is selected as the first domain image, and a male image is selected as the second domain image. If the image style needs to be converted from a photo to an oil painting, male photos, female photos, landscape photos, and other photos are selected as the first domain image, and male oil paintings, female oil paintings, landscape oil paintings, and other oil paintings are selected as the second domain image accordingly. Image style transfer can also be a variety of combinations, for example, oil painting to photo, photo to Chinese painting, Chinese painting to photo, Monet to Picasso, etc. You only need to use the original type of image as the first domain image and the target type of image as the second domain image to achieve image conversion between any types.
[0033] Step 104: input the first domain image of the sample into a generation module to obtain a pseudo image output by the generation module and a reference image output by the network module.
[0034] In this embodiment, the generation module may convert the input first domain image into a pseudo image of the second domain. Figure 2 As shown, a photo face image is input and a cute face image is output.
[0035] In some optional implementations of this embodiment, in order to further improve the quality of the image formed by the network module, the intermediate layer features of the generation module can be extracted, and the extracted intermediate layer features can be input into the network module so that the network module outputs a new reference image; and the new reference image is used to replace the reference image.
[0036] In this optional implementation, the intermediate layer features of the generation module may be the intermediate layer features of the decoder in the generation module, and the network module fuses the received intermediate layer features to generate a new reference image. Generally, the resolution of the intermediate layer features should be the same as the resolution of the network module. When the resolution of the intermediate layer features is different from the resolution of the network module, deconvolution or upsampling can be used to ensure that the two are of the same size to achieve fusion.
[0037] In this optional implementation, fusion refers to splicing the intermediate layer features with the features of the network module. For example, the intermediate layer features are 32 channels, and the features of the network module are also 32 channels, and 64 channels are obtained after fusion.
[0038] In this optional implementation, by extracting the intermediate features of the generation module, the features of the network module are enhanced, which facilitates the network module to generate clearer reference images and improves the refinement of the overall and local features of the reference image generated by the network module.
[0039] Step 105: input the second domain image, the pseudo image and the reference image of the sample into the discriminant network of the generative adversarial network.
[0040] In this embodiment, except that the number of discrimination times of the discriminant network is increased, the function of the discriminant network of the present application is the same as that of the classic discriminant network. If the discriminant network determines that the pseudo image output by the above-mentioned generation module is the input second domain image, it can output 1; if it determines that the pseudo image output by the above-mentioned generation module is not the input second domain image (from real data), it can output 0. If the discriminant network determines that the reference image output by the above-mentioned network module is the input second domain image, it can output 1; if it determines that the reference image output by the above-mentioned network module is not the input second domain image (from real data), it can output 0. It should be noted that the discriminant network can also output other numerical values based on pre-settings, not limited to 1 and 0, such as confidence score values. Furthermore, the L1 loss value or L2 loss value of the pseudo image of the sample and the second domain image of the sample can be calculated, and the L1 loss value or L2 loss value of the reference image of the sample and the second domain image of the sample can be calculated.
[0041] In this embodiment, optionally, the network module can be a module, for example, a convolutional network composed of 7 convolutional layers.
[0042] In some optional implementations of this embodiment, the network module may further include a plurality of cascade-connected convolution modules, each of which includes a set number of convolution layers connected in sequence. Figure 2 As shown, the network module w includes: 5 cascade-connected convolution modules (w1, w2, w3, w4, w5).
[0043] In this embodiment, each cascaded network module has a similar structure, and each network module outputs a second domain image. As the depth of the cascade increases, the image output by the subsequent network modules has a higher degree of refinement. Figure 2 In all cascaded network modules, the input of each network module is different, and the input of the subsequent network module is the output of the previous network module. The subsequent network module makes fine adjustments based on the image generated by the previous network module to generate a more refined image than the previous network module.
[0044] In this embodiment, the number of cascaded network modules is generally set based on experience. Figure 2 In the example, the number of cascaded network modules can be 5. It should be noted that if there are too few cascaded network modules, the resulting image will not be fine enough. If there are too many cascaded network modules, the computational complexity of the adversarial generative network will increase, which will slow down the training speed. Moreover, the fineness of the output image will not increase with the increase of the number of cascaded network modules. After the number of cascaded network modules reaches a certain number, the benefit of increasing the number of cascaded modules is minimal.
[0045] In this optional implementation, the convolution module is a module composed of a set number of convolution layers, and the set number can be set according to demand, for example, the set number is 7.
[0046] In this optional implementation, the cascaded network module is largely responsible for obtaining the most abstract information in the image generated by the generation module, thereby making better inferences about the content of the image and ensuring the reliability of information extraction from the image.
[0047] In this embodiment, optionally, the output images of all cascaded convolution modules can be used as reference images and input into the discriminant network of the generative adversarial network.
[0048] In some optional implementations of the present embodiment, the above-mentioned inputting the reference image into the discriminant network of the generative adversarial network includes: inputting the image output by each convolution module into the discriminant network of the generative adversarial network, so that the discriminant network classifies the images output by each convolution module.
[0049] like Figure 2As shown in the figure, the output of each convolution module is connected to the discriminant network D, so that the output images of each convolution module will enter the discriminant network, and the discriminant network will discriminate the pictures of the second domain after the output of each convolution module is converted one by one. The discrimination result can be multiple confidence scores, and then all the confidence scores are averaged to obtain a final confidence score, which is the confidence score of the reference image.
[0050] In this optional implementation, the structure of the discriminant network of the adversarial generative network is not improved, but the number of discriminations of the discriminant network is increased. By averaging all the discrimination results, a discrimination result with higher reference value is obtained.
[0051] In this optional implementation, each network module can maintain a similar structure, and the image output by each network module is sent to the discriminant network for discrimination (equivalent to relay supervision). The image output by the lower-level network module may be rough and not very fine, but through the cascade of multiple network modules, each network module is further improved on the basis of the previous network module, so that the image output by the later network module is more refined in terms of completeness and detail clarity.
[0052] Step 106 , in response to determining that the generative adversarial network meets the training completion condition, using the generative adversarial network as an image generation model.
[0053] In this embodiment, the training completion condition includes at least one of the following: the number of training iterations reaches a predetermined iteration threshold, after the second domain image, the pseudo image and the reference image of the sample are input into the discriminant network of the generative adversarial network together, the calculated loss value is less than the predetermined loss value threshold, and the discrimination accuracy of the discriminant network is within a predetermined range. For example, the training iterations reach 5,000 times. The loss value is less than 0.05, and the discrimination accuracy of the discriminant network reaches 50%. After the training is completed, the generative adversarial network is used as an image generation model. Setting the training completion condition can speed up the convergence of the image generation model.
[0054] In some optional implementations of this embodiment, after step 106, it may also include: an adjustment step: if the generative adversarial network does not meet the training completion conditions, then adjust the relevant parameters in the generative adversarial network so that the loss value converges, and continue to execute steps 103-106 based on the adjusted generative adversarial network.
[0055] In this embodiment, if the training is not completed, the parameters of the generating network or the discriminating network are adjusted to make the loss value converge. The parameters of the discriminating network can be kept unchanged, and steps 103-106 and the adjustment steps are repeatedly performed to adjust the parameters of the corresponding generating network so that the loss value gradually decreases until it stabilizes. Then, the parameters of the generating network are kept unchanged, and steps 103-106 and the adjustment steps are repeatedly performed to adjust the parameters of the discriminating network so that the loss value gradually increases until it stabilizes. The parameters of the generating network and the parameters of the discriminating network are alternately trained until the loss value converges.
[0056] The image generation model training method provided in this embodiment not only distinguishes true and false images generated by the generation module, but also distinguishes true and false images output by the network module. After adding the network module, the quality of the images generated by the network is improved, not only the integrity of the images is improved, but also the clarity and detail richness of the images are improved.
[0057] The image generation model generated by the image generation model training method provided by the present embodiment is not only applicable to picture translation frameworks such as cycle generative adversarial networks (cycleGAN) and pix2pix (predict from pixels to pixels), but is also widely applicable to a very broad field related to picture generation by adversarial generative networks. Even a major change in the algorithm is not required. Only a few network modules need to be added after the decoder of the generation module and supervised separately, which can greatly improve the effect of the final generated picture. The generated picture has fewer defects, and the completeness, clarity, and authenticity are significantly improved.
[0058] See also Figure 3 , which shows a process 300 of an embodiment of the image generation method provided by the present application. The image generation method may include the following steps:
[0059] Step 301: Obtain an image to be processed.
[0060] In this embodiment, the execution subject of the image generation method can obtain the image to be processed in a variety of ways. For example, the execution subject can obtain the image to be processed stored in the database server through a wired connection or a wireless connection. For another example, the execution subject can also receive the image to be processed collected by a terminal (such as a terminal connected to the database server) or other devices.
[0061] In this embodiment, the image to be processed may be a color image and / or a grayscale image, etc. The format of the image to be processed is not limited in this application.
[0062] Step 302: input the image to be processed into the image generation model, and output the generated image.
[0063] In this embodiment, the execution subject can input the image to be processed obtained in step 301 into the image generation model, so that the image generation model outputs the generated image. The image generation model trained and generated by steps 101-106 can convert the image from the first domain to the second domain. Using the image generation model trained by steps 101-106, inputting a face image in photo format can convert it into a Q-cute face image.
[0064] In this embodiment, the image generation model can be adopted as described above Figure 1 The specific generation process can be found in Figure 1 The relevant description of the embodiments will not be repeated here.
[0065] It should be noted that the image generation method of this embodiment can be used to test the image generation model generated by the above embodiments. Then, the image generation model can be continuously optimized according to the conversion result. This method can also be a practical application method of the image generation model generated by the above embodiments. Using the image generation model generated by the above embodiments to perform image generation helps to improve the fineness of the generated image.
[0066] Further references Figure 4 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an image generation model training device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0067] like Figure 4As shown, the image generation model training device 400 provided in this embodiment includes: a sample acquisition unit 401, a network acquisition unit 402, a selection unit 403, a obtaining unit 404, an input unit 405, and a model generation unit 406. The sample acquisition unit 401 can be configured to obtain a preset sample set, wherein the sample set includes at least one sample, and the sample includes a first domain image and a second domain image. The network acquisition unit 402 can be configured to obtain a pre-established generative adversarial network, wherein the generative network of the generative adversarial network includes a generation module and a network module, and the network module is used to improve the fineness of the image output by the generation module. The selection unit 403 can be configured to select a sample from the sample set. The obtaining unit 404 can be configured to input the first domain image of the sample into the generation module to obtain a pseudo image output by the generation module and a reference image output by the network module. The input unit 405 can be configured to input the second domain image, the pseudo image and the reference image of the sample into the discriminant network of the generative adversarial network. The model generation unit 406 may be configured to use the generative adversarial network as an image generation model in response to determining that the generative adversarial network meets the training completion condition.
[0068] In this embodiment, in the image generation model training device 400, the specific processing of the sample acquisition unit 401, the network acquisition unit 402, the selection unit 403, the obtaining unit 404, the input unit 405, and the model generation unit 406 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104, step 105, and step 106 in the corresponding embodiments are not repeated here.
[0069] In some optional implementations of this embodiment, the apparatus 400 further includes: an extraction unit (not shown in the figure), a configuration unit (not shown in the figure), and a replacement unit (not shown in the figure). The extraction unit may be configured to extract the intermediate layer features of the generation module. The configuration unit may be configured to input the extracted intermediate layer features into the network module so that the network module outputs a new reference image. The replacement unit may be configured to replace the reference image with a new reference image.
[0070] In some optional implementations of this embodiment, the above-mentioned network module includes a plurality of cascade-connected convolution modules, and each convolution module includes a set number of convolution layers connected in sequence.
[0071] In some optional implementations of this embodiment, the above-mentioned configuration unit is further configured to: input the image output by each convolution module into the discriminant network of the generative adversarial network, so that the discriminant network classifies the images output by each convolution module.
[0072] In some optional implementations of this embodiment, the training completion condition includes at least one of the following:
[0073] The number of training iterations reaches a predetermined iteration threshold, the discrimination accuracy of the discriminant network is within a predetermined range, and after the second domain image, the pseudo image, and the reference image of the sample are input into the discriminant network of the generative adversarial network together, the calculated loss value is less than the predetermined loss value threshold.
[0074] The image generation model generated by the image generation model training device provided by this embodiment is not only suitable for picture translation frameworks such as cycle generative adversarial networks (cycleGAN) and pix2pix (predict from pixels to pixels), but is also widely applicable to a very broad field related to picture generation by adversarial generative networks. Even a major change in the algorithm is not required. Only a few network modules need to be added after the decoder of the generation module and supervised separately, which can greatly improve the effect of the final generated picture. The generated picture has fewer defects, and the completeness, clarity, and authenticity are significantly improved.
[0075] Continue to see Figure 5 , as a response to the above Figure 3 To implement the method shown in the figure, the present application provides an embodiment of an image generation device. Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0076] like Figure 5 As shown, the image generation device 500 of this embodiment may include: an image acquisition unit 501, configured to acquire an image to be processed. An image generation unit 502, configured to input the image to be processed into the image generation unit 502 as described above. Figure 1 or Figure 3 In the image generation model generated by the method described in the embodiment, the generated image is output.
[0077] It is understood that the units described in the device 500 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.
[0078] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0079] like Figure 6, is a block diagram of an electronic device according to an image generation model training method or an image generation method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0080] like Figure 6 As shown, the electronic device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.
[0081] The memory 602 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the image generation model training method or image generation method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the image generation model training method or image generation method provided in the present application.
[0082] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the image generation model training method in the embodiment of the present application (for example, the attached Figure 4The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, the image generation model training method in the above method embodiment is implemented.
[0083] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of an electronic device according to the image generation model training method, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely disposed relative to the processor 601, and these remote memories may be connected to the electronic device of the image generation model training method or the image generation method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0084] The electronic device of the image generation model training method or the image generation method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus 605 or other means. Figure 6 The connection via bus 605 is taken as an example.
[0085] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the image generation model training method or the image generation method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0086] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0087] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0089] The systems and techniques described herein may be implemented in a computing system including a backend component (e.g., as a data server), or a computing system including a middleware component (e.g., an application server), or a computing system including a frontend component (e.g., a user computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network. A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS, Virtual Private Server) services. The server can also be a server of a distributed system, or a server combined with a blockchain.
[0090] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0091] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A method for training an image generation model, the method comprising: Acquire a preset sample set, wherein the sample set includes at least one sample, the sample includes a first domain image and a second domain image, and the domain refers to a category of an image; Acquire a pre-established generative adversarial network, wherein the generative network of the generative adversarial network includes a generation module and a network module, and the network module is used to improve the fineness of the image output by the generation module; The following training steps are performed: selecting a sample from the sample set; inputting a first domain image of the sample into the generation module to obtain a pseudo image output by the generation module and a reference image output by the network module; inputting a second domain image of the sample, the pseudo image and the reference image together into a discriminant network of the generative adversarial network; in response to determining that the generative adversarial network meets the training completion condition, using the generative adversarial network as an image generation model.
2. The method according to claim 1, further comprising: Extracting intermediate layer features of the generation module, and inputting the extracted intermediate layer features into the network module, so that the network module outputs a new reference image; The reference image is replaced with a new reference image.
3. The method according to claim 1, wherein: The network module includes a plurality of cascade-connected convolution modules, and each convolution module includes a set number of convolution layers connected in sequence.
4. The method according to claim 3, wherein inputting the reference image into the discriminative network of the generative adversarial network comprises: The image output by each convolution module is input into the discriminant network of the generative adversarial network, so that the discriminant network classifies the images output by each convolution module.
5. The method according to any one of claims 1 to 4, wherein: The training completion condition includes at least one of the following: The number of training iterations reaches a predetermined iteration threshold, the discrimination accuracy of the discriminant network is within a predetermined range, and after the second domain image of the sample, the pseudo image, and the reference image are input together into the discriminant network of the generative adversarial network, the calculated loss value is less than the predetermined loss value threshold.
6. A method for generating an image, the method comprising: Get the image to be processed; The image to be processed is input into an image generation model generated by the method according to any one of claims 1 to 5, and the generated image is output.
7. An image generation model training device, the device comprising: A sample acquisition unit is configured to acquire a preset sample set, wherein the sample set includes at least one sample, the sample includes a first domain image and a second domain image, and the domain refers to a category of an image; A network acquisition unit is configured to acquire a pre-established generative adversarial network, wherein the generative network of the generative adversarial network includes a generation module and a network module, and the network module is used to improve the fineness of the image output by the generation module; A selection unit, configured to select a sample from the sample set; an obtaining unit configured to input the first domain image of the sample into the generating module to obtain a pseudo image output by the generating module and a reference image output by the network module; An input unit, configured to input the second domain image of the sample, the pseudo image and the reference image together into a discriminative network of the generative adversarial network; The model generation unit is configured to use the generative adversarial network as an image generation model in response to determining that the generative adversarial network meets the training completion condition.
8. The device according to claim 7, further comprising: an extraction unit, configured to extract intermediate layer features of the generation module; A configuration unit, configured to input the extracted intermediate layer features into the network module so that the network module outputs a new reference image; A replacement unit is configured to replace the reference image with a new reference image.
9. The device according to claim 7, wherein: The network module includes a plurality of cascade-connected convolution modules, and each convolution module includes a set number of convolution layers connected in sequence.
10. According to the device of claim 8, the configuration unit is further configured to: input the image output by each convolution module into the discriminant network of the generative adversarial network, so that the discriminant network classifies the images output by each convolution module.
11. The device according to any one of claims 7 to 10, wherein: The training completion condition includes at least one of the following: The number of training iterations reaches a predetermined iteration threshold, the discrimination accuracy of the discriminant network is within a predetermined range, and after the second domain image of the sample, the pseudo image, and the reference image are input together into the discriminant network of the generative adversarial network, the calculated loss value is less than the predetermined loss value threshold.
12. An image generating device, the device comprising: An image acquisition unit, configured to acquire an image to be processed; The image generation unit is configured to input the image to be processed into an image generation model generated by the method according to any one of claims 1 to 5, and output the generated image.
13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.
