Training method of image generation model, image generation method, device and equipment

Abstract

The application provides a training method and an image generation method and device and equipment of an image generation model, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises the following steps: inputting a sample natural image into a first to-be-trained generator to obtain a generated stylized image, and inputting the generated stylized image into a second to-be-trained generator to obtain a reconstructed natural image; inputting a sample stylized image into the second to-be-trained generator to obtain a generated natural image, and inputting the generated natural image into the first to-be-trained generator to obtain a reconstructed stylized image; constructing a target loss function according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image, and training the first to-be-trained generator and the second to-be-trained generator, and the first to-be-trained generator at the end of the training is used as an image generation model. The application can improve the robustness of the image generation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a training method of image generation model, an image generation method, device and equipment. BACKGROUND

[0002] The algorithm for stylizing natural images in the prior art includes a feature fusion-based method and a generative adversarial network-based method. However, in the feature fusion-based method, a single image cannot accurately describe the latent features of a class of style images, resulting in the inability to transfer structural texture style information. In the generative adversarial network-based method, the learning process of the data sets of the two image domains does not consider the case where the information amounts of the two image domains are not equal. Therefore, the quality of the images generated based on the above two methods is poor. SUMMARY

[0003] In order to solve the problem that the model for stylizing natural images in the prior art cannot generate high-quality images, the present application provides a training method of image generation model, an image generation method, device and equipment:

[0004] According to a first aspect of the present application, a training method of image generation model is provided, comprising:

[0005] obtaining a sample natural image and a sample stylized image;

[0006] inputting the sample natural image into a first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and inputting the generated stylized image into a second to-be-trained generator for natural generation processing to obtain a reconstructed natural image;

[0007] inputting the sample stylized image into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and inputting the generated natural image into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image;

[0008] constructing a target loss function according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image;

[0009] training the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training;

[0010] wherein the first to-be-trained generator at the end of the training is used as an image generation model.

[0011] According to a second aspect of the present application, a training model of image generation model is provided, comprising:

[0012] The first obtaining module is configured to obtain a sample natural image and a sample stylized image.

[0013] The first processing module is configured to input the sample natural image into the first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and input the generated stylized image into the second to-be-trained generator for natural generation processing to obtain a reconstructed natural image.

[0014] The second processing module is configured to input the sample stylized image into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and input the generated natural image into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image.

[0015] The constructing module is configured to construct a target loss function according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image.

[0016] The first training module is configured to train the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training.

[0017] The first to-be-trained generator at the end of the training is used as the image generation model.

[0018] On the other hand, the constructing module comprises:

[0019] The edge feature extraction submodule is configured to input the sample natural image, the generated stylized image and the reconstructed natural image into an edge extractor for edge feature extraction processing to obtain a first edge feature corresponding to the sample natural image, a second edge feature corresponding to the generated stylized image and a third edge feature corresponding to the reconstructed natural image. The parameters of the edge extractor remain unchanged during the entire training process.

[0020] The first constructing submodule is configured to construct a first loss function according to the first edge feature, the second edge feature and the third edge feature.

[0021] The second constructing submodule is configured to construct a second loss function according to difference information of the sample stylized image and the reconstructed stylized image.

[0022] The third constructing submodule is configured to construct the target loss function according to the first loss function and the second loss function.

[0023] On the other hand, the first constructing submodule is configured to determine a first sub-loss value according to a difference between the first edge feature and the second edge feature.

[0024] determine a second sub-loss value according to a difference between the first edge feature and the third edge feature.

[0025] determine the first loss function according to a sum value of the first sub-loss value and the second sub-loss value.

[0026] In another aspect, the third construction sub-module comprises:

[0027] The first input and output unit is configured to input the generated stylized image and the sample stylized image into the first discriminator to obtain a first discrimination value corresponding to the generated stylized image and a second discrimination value corresponding to the sample stylized image.

[0028] The first determination unit is configured to determine a third loss function according to a sum value of a difference between the first label value corresponding to the generated stylized image and the first discrimination value and a difference between the second label value corresponding to the sample stylized image and the second discrimination value.

[0029] The second input and output unit is configured to input the generated natural image and the sample natural image into the second discriminator to obtain a third discrimination value corresponding to the generated natural image and a fourth discrimination value corresponding to the sample natural image.

[0030] The second determination unit is configured to determine a fourth loss function according to a sum value of a difference between the first label value corresponding to the generated natural image and the third discrimination value and a difference between the second label value corresponding to the sample natural image and the fourth discrimination value.

[0031] The construction unit is configured to construct a target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function.

[0032] In another aspect, the training device of the image generation model further comprises a second training module.

[0033] The second training module is configured to train the first discriminator and the second discriminator.

[0034] Training the first discriminator comprises:

[0035] inputting the generated stylized image and the sample stylized image into the first to-be-trained discriminator to obtain a first candidate discrimination value corresponding to the generated stylized image and a second candidate discrimination value corresponding to the sample stylized image;

[0036] determining a fifth loss function according to a sum value of a difference between the second label value corresponding to the generated stylized image and the first candidate discrimination value and a difference between the first label value corresponding to the sample stylized image and the second candidate discrimination value.

[0037] adjusting model parameters of the first to-be-trained discriminator based on the fifth loss function until a second preset training end condition is met to end the training, and obtaining the first discriminator.

[0038] Training the second discriminator comprises:

[0039] input the generated natural image and the sample natural image into the second to-be-trained discriminator to obtain a third candidate discrimination value corresponding to the generated natural image and a fourth candidate discrimination value corresponding to the sample natural image;

[0040] determine a sixth loss function according to a sum value of a difference value between the second label value corresponding to the generated natural image and the third candidate discrimination value and a difference value between the first label value corresponding to the sample natural image and the fourth candidate discrimination value;

[0041] adjust the model parameters of the second to-be-trained discriminator based on the sixth loss function until a third preset training end condition is met to end the training, and obtain a second discriminator.

[0042] On the other hand, the first to-be-trained generator comprises a plurality of to-be-trained sub-level generators;

[0043] an edge feature extraction submodule configured to obtain a first multi-scale image corresponding to the sample natural image and a multi-scale sub-generated stylized image corresponding to the first multi-scale image; the scale of the first multi-scale image is equal to the number of levels of the to-be-trained sub-level generator;

[0044] input the first multi-scale image, the multi-scale sub-generated stylized image and the reconstructed natural image into the edge extractor respectively for edge feature extraction processing to obtain a first multi-scale edge feature corresponding to the first multi-scale image, a second multi-scale edge feature corresponding to the multi-scale sub-generated stylized image, and a third edge feature corresponding to the reconstructed natural image;

[0045] a first construction submodule configured to construct a first loss function according to the first multi-scale edge feature, the second multi-scale edge feature and the third edge feature.

[0046] On the other hand, the first input and output unit is configured to obtain a second multi-scale image corresponding to the sample natural image; the second multi-scale image is an image obtained by performing down-sampling processing on the sample natural image;

[0047] input the first multi-scale image and the second multi-scale image into the first discriminator to obtain a first reference discrimination value corresponding to each image in the first multi-scale image and a second reference discrimination value corresponding to each image in the second multi-scale image;

[0048] a first determination unit configured to determine a third loss function according to a sum value of a difference value between a first label value corresponding to each image in the first multi-scale image and the first reference discrimination value and a difference value between a second label value corresponding to each image in the second multi-scale image and the second reference discrimination value.

[0049] On the other hand, the edge feature extraction submodule is configured to perform down-sampling processing on the sample natural image to obtain a down-sampled image corresponding to the down-sampling processing;

[0050] The down-sampled image is up-sampled to obtain an up-sampled image corresponding to the current sampling processing; the up-sampled image and the sample natural image have the same resolution;

[0051] According to the difference information of the sample natural image and the up-sampled image, a residual image corresponding to the current sampling processing is determined;

[0052] The down-sampled image corresponding to the current sampling processing is taken as a sample natural image to be processed to obtain a set of residual images corresponding to the multiple sampling processing;

[0053] According to the down-sampled image corresponding to the last sampling processing in the multiple sampling processing and the set of residual images corresponding to the multiple sampling processing, a first multi-scale image is constructed;

[0054] The images in the first multi-scale image are respectively input into a plurality of to-be-trained sub-level generators to obtain multi-scale sub-generated stylized images corresponding to the first multi-scale image.

[0055] According to a third aspect of the present application, an image generation method is provided, comprising:

[0056] Obtaining a natural image to be processed;

[0057] Inputting the natural image to be processed into an image generation model to output a target stylized image corresponding to the natural image to be processed; the image generation model is a model obtained based on the training method of the image generation model of the first aspect of the present application.

[0058] According to a fourth aspect of the present application, an image generation device is provided, comprising:

[0059] The second acquisition module is configured to acquire a natural image to be processed;

[0060] The input and output module is configured to input the natural image to be processed into an image generation model to output a target stylized image corresponding to the natural image to be processed; the image generation model is a model obtained based on the training method of the image generation model of the first aspect of the present application.

[0061] According to a fifth aspect of the present application, an electronic device is provided, which comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the training method of the image generation model of the first aspect of the present application.

[0062] According to a sixth aspect of the present application, a computer storage medium is provided, which stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the training method of the image generation model of the first aspect of the present application.

[0063] According to a seventh aspect of the present application, a computer program product is provided, the computer program product comprising at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the training method of the image generation model of the first aspect of the present application.

[0064] The training method of the image generation model, the image generation method, the device and the equipment provided by the embodiments of the present application have the following technical effects: a sample natural image and a sample stylized image are obtained; the sample natural image is input into a first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and the generated stylized image is input into a second to-be-trained generator for natural generation processing to obtain a reconstructed natural image; the sample stylized image is input into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and the generated natural image is input into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image; a target loss function is constructed according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image; the first to-be-trained generator and the second to-be-trained generator are trained based on the target loss function until a first preset training end condition is met to end the training; and the first to-be-trained generator at the end of the training is taken as the image generation model. By constructing an optimization target through mutual conversion of natural images and stylized images, the robustness of the image generation model can be improved, and the quality and resolution of the generated stylized image can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 is a schematic diagram of an application environment provided by the embodiments of the present application;

[0067] Figure 2 is a flowchart of a training method of an image generation model provided by the embodiments of the present application;

[0068] Figure 3 is a framework diagram of a training method of an image generation model provided by the embodiments of the present application;

[0069] Figure 4 is a schematic diagram of determining a multi-scale sub-generated stylized image corresponding to a first multi-scale image provided by the embodiments of the present application;

[0070] Figure 5 is a structural schematic diagram of an image generation device provided by an embodiment of the present application.

[0071] Figure 6 is a flowchart of an image generation method provided by an embodiment of the present application.

[0072] Figure 7 is a schematic diagram of an image generation method provided by an embodiment of the present application.

[0073] Figure 8 is a structural schematic diagram of an image generation device provided by an embodiment of the present application.

[0074] Figure 9 is a hardware structural schematic diagram of an electronic device for implementing a training method of an image generation model or an image generation method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0075] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0076] The "embodiments" referred to herein can include specific features, structures or characteristics that can be included in at least one implementation of the present application. In the description of the embodiments of the present application, it should be understood that the terms "first", "second" and "third" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" and "third" can be explicitly or implicitly included one or more of the features. Moreover, the terms "first", "second" and "third" are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include", "have" and "be" and any variations thereof are intended to cover non-exclusive inclusion.

[0077] It can be understood that in the detailed description of the present application, data related to image data and the like is required to obtain user permission or consent when the above embodiments of the present application are applied to specific products or technologies, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0078] Embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, computer vision, assisted driving, etc.

[0079] Among them, artificial intelligence (Artificial Intelligence, AI) is to use digital computers or digital computer controlled machine simulation, extension and expansion of human intelligence, perception of environment, acquisition of knowledge and use of knowledge to obtain the best results of theory, method, technology and application system. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. several directions.

[0080] Computer vision technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further to do image processing, so that the computer processing becomes more suitable for human eye observation or image transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0081] Machine learning (Machine Learning, ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a branch of computer science that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and example-based learning.

[0082] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0083] Generative Adversarial Networks (GAN): a deep learning model including two generators, two discriminators and an edge extractor, which produces a fairly good output through mutual game learning of the generators and the discriminators;

[0084] Natural image: an image obtained by shooting objects such as scenes, objects and figures in the physical world through a camera;

[0085] Laplacian image pyramid: sequentially performing smoothing (optional), down-sampling, up-sampling on an image, obtaining a residual image by subtracting the up-sampled image from the initial image, and performing the above operations on the down-sampled image to obtain multiple levels of residual images and a minimum resolution image, which together constitute a Laplacian image pyramid;

[0086] L1 Loss and L2 Loss: L1 or L2 norm of two tensors (multi-dimensional matrices) as a distance measure.

[0087] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment provided by the embodiments of the present application, which can include a collection device 10 and a server 20. The collection device 10 and the server 20 can be directly or indirectly connected through wired or wireless communication.

[0088] In some possible embodiments, the collection device 10 can send sample natural image data and sample stylized image data to the server 20. The server can provide a training image generation model service to construct a target loss function through mutual conversion of natural images and stylized images.

[0089] The collection device 10 can include at least one of a camera, a camera, a camera and other hardware devices that can shoot images. The server 20 can be a standalone physical server, or a service cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services. Among them, the server can include a network communication unit, a processor and a memory and the like. The server can provide an image back transmission service.

[0090] In some possible implementation manners, the collection device 10 and the server 20 can be node devices in a blockchain system, and can share the acquired and generated information to other node devices in the blockchain system, so as to realize information sharing between the node devices. The node devices in the blockchain system can be configured with the same blockchain, the blockchain is composed of a plurality of blocks, and the blocks adjacent in front and back have an association relationship, so that when the data in any block is tampered with, it can be detected through the next block, thereby avoiding the data in the blockchain being tampered with, and ensuring the safety and reliability of the data in the blockchain.

[0091] There is no algorithm in the prior art that is specifically used for paper-cut stylization of natural images. The closest to the paper-cut stylization task is artistic image stylization. There are two main types of algorithms for artistic image stylization of natural images. One type is a feature fusion-based method, which uses a feature extraction network to extract multi-level features of natural images and style images, then performs fusion processing on the multi-level features, and inputs the fusion processing result into a neural network for decoding processing to obtain an image based on the semantic features of the natural image and the style features of the style image. Generally, the feature extraction network directly uses a feature extraction network model that has been trained on a large visual database ImageNet classification dataset for visual object recognition software research. Based on the feature fusion method, the neural network cannot learn the style common to a certain type of image data, and the style feature of the output image is often determined by a single image. However, since a single image cannot accurately describe the potential high-dimensional style features of a class of styles, the semantic information of the style image is leaked into the result, while the structural and texture style information cannot be transferred. For the paper-cut style transfer task, its characteristics are that the generated image color style changes less (the generated image substrate only contains two colors), and the generated texture style is more (adjusting the coloring of the texture while ensuring the semantic information of the result). This method only combines the features of natural images and style images and then decodes to obtain an image, and does not have additional creative ability to obtain a visually pleasing effect. Therefore, this method cannot meet the demand of paper-cut stylization.

[0092] Another type is a method based on a generative adversarial network (GAN), which adopts a generative adversarial idea and contains a generative network and a discriminative network. The generative network replicates the projection from one image domain to another image domain, and the discriminative network is used to judge whether the generated result of the generative network is realistic enough. A relatively classic one is a Cycle Generative Adversarial (CycleGAN) network that realizes the function of image segmentation conversion. The CycleGAN network converts images of two styles or two image domains into each other by using two generators and two discriminators, and establishes the relationship between the real images and the generated images of a certain image domain by using cycle consistency and performs optimization. The method based on the generative adversarial network obtains a mapping from one image domain to another image domain by learning the data sets of the two image domains, and can meet the demand of paper-cut stylization compared with the feature fusion method. However, the method does not consider the case that the information amount of the two image domains is not equal. In paper-cut stylization, converting a natural image into a paper-cut image is a process of reducing the information amount, which is theoretically easier. However, converting a paper-cut image into a natural image is a process of increasing the information amount, which requires the generator to have stronger generation ability. The mismatched optimization target will affect the overall performance of the algorithm. Therefore, the goal of the present application is to design a more matched optimization target to enhance the generation ability of the network so that it can generate more satisfactory images.

[0093] The following describes a specific embodiment of a training method of an image generation model according to the present application, Figure 2 is a flowchart of a training method of an image generation model according to an embodiment of the present application, Figure 3 is a framework diagram of a training method of an image generation model according to an embodiment of the present application. The present specification provides method operation steps as shown in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one of many execution orders, and does not represent the only execution order. In actual execution, the method order shown in the embodiments or the drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment).

[0094] Specifically as shown in Figure 2 and 3 , the training method of the image generation model can include:

[0095] S201: Obtain a sample natural image and a sample stylized image.

[0096] In the embodiments of the present application, the sample natural image A and the sample stylized image B collected by the collection device can be obtained. The sample natural image A can be an image obtained by a camera shooting a scene, an object, a person or the like in a physical world, and the sample stylized image B can be a paper-cut image. The sample natural image A and the sample stylized image B can be images containing different types of objects. For example, the sample natural image A can be a natural image containing an eagle, and the sample stylized image B can be a paper-cut image containing a building.

[0097] In S203, the sample natural image is input into the first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and the generated stylized image is input into the second to-be-trained generator for natural generation processing to obtain a reconstructed natural image.

[0098] In the embodiments of the present application, the entire training process can include five sub-networks, i.e., the first generator G_A, the second generator G_B, the first discriminator D_A and the second discriminator D_B, and an edge extractor E. The edge extractor E can be any pre-trained edge extractor, and the gradient is not updated in the entire training process.

[0099] In a single training process, the sample natural image A can be first input into the first to-be-trained generator G_A to output a generated stylized image Fake B. Then the generated stylized image Fake B can be input into the second to-be-trained generator G_B to output a reconstructed natural image Rec A. The sample natural image A, the generated stylized image Fake B and the reconstructed natural image Rec A can be images containing the same type of object. For example, the sample natural image A, the generated stylized image Fake B and the reconstructed natural image Rec A can be images containing an eagle.

[0100] To provide generators with stronger generation capability, each generator can be designed to include multiple sub-level generators to be trained, for example, the first generator can include three sub-level generators to be trained, i.e., a first sub-level generator to be trained G_1, a second sub-level generator to be trained G_2, and a third sub-level generator to be trained G_3. Each sub-level generator to be trained can be composed of a basic structure residual block ResBlock in multiple residual networks (Residual Network, ResNet), constituting a network of an Encoder-Decoder overall structure. Each sub-level generator to be trained can also adopt a semantic segmentation network structure, which is not limited in the embodiments of the present application. Each sub-level generator to be trained can adopt a step-by-step training manner, i.e., after the current level is trained, the parameters are fixed and not updated, and the input is provided for the training of the next level. Each generator can form a complete generative adversarial network with a corresponding discriminator, for example, the first generator G_A and the first discriminator D_A, the second generator G_B and the second discriminator D_B. The sub-level generators to be trained in each generator can share a discriminator, for example, the first sub-level generator to be trained G_1, the second sub-level generator to be trained G_2, and the third sub-level generator to be trained G_3 share the first discriminator. The sub-level generators to be trained in each generator can also have independent discriminators, such as the first sub-level generator to be trained G_1 and D_1 adversarial learning, the second sub-level generator to be trained G_2 and D_2 adversarial learning, and the third sub-level generator to be trained G_3 and D_3 adversarial learning. Each discriminator can also adopt a basic structure ResBlock in multiple ResNets to constitute, and constantly reduce the dimension of the features to facilitate the difference with the label value, i.e., the true value constant (0 or 1), to determine the loss function. By using multiple-level generators to gradually generate and modify the results of the previous level, high-quality and high-resolution paper-cut images can be generated.

[0101] In the embodiments of the present application, after obtaining the sample natural image, a Laplacian image pyramid corresponding to the sample natural image can be constructed as a whole input of the generator. After outputting the generated stylized image, a Laplacian image pyramid corresponding to the output generated stylized image can be constructed. After obtaining the sample natural image, the sample natural image can be down-sampled to obtain a down-sampled image corresponding to the current sampling processing, and then the down-sampled image can be up-sampled to obtain an up-sampled image corresponding to the current sampling processing. The resolution of the up-sampled image is the same as that of the sample natural image. Then, according to the difference information between the sample natural image and the up-sampled image, a residual image corresponding to the current sampling processing can be determined. Subsequently, the down-sampled image corresponding to the current sampling processing can be taken as the sample natural image for sampling processing to obtain a set of residual images corresponding to multiple sampling processing. According to the down-sampled image corresponding to the last sampling processing in the multiple sampling processing and the set of residual images corresponding to the multiple sampling processing, a first multi-scale image, i.e., a Laplacian image pyramid corresponding to the sample natural image, can be constructed. By constructing the Laplacian image pyramid, the information of the image can be enriched, and a high-quality, high-resolution paper-cut image can be generated.

[0102] The following is described by taking constructing a Laplacian image pyramid corresponding to a sample natural image as an example. "1x" can represent that the resolution of the image is the same as that of the original image, "2x" can represent that the resolution of the image is 1 / 2 of that of the original image, and so on. "Up-sampling 2x" can represent that the image is up-sampled by 2 times in length and width. In actual application, if the first generator includes three to-be-trained sub-level generators, i.e., a first to-be-trained sub-level generator G_1, a second to-be-trained sub-level generator G_2, and a third to-be-trained sub-level generator G_3, the sample natural image can be represented by a matrix. It can be assumed that the sample natural image is Image 1x. The Image 1x is subjected to low-pass filtering processing, and after the low-pass filtering processing is completed, a first down-sampled image Image 2x is obtained by 2 times down-sampling. Then, the first down-sampled image Image 2x can be subjected to 2 times up-sampling to obtain a first up-sampled image Image 2x_up, and a first residual image Res 1x can be determined by the difference between the corresponding values in the matrix corresponding to the sample natural image Image 1x and the matrix corresponding to the first up-sampled image Image 2x_up. The first residual image Res 1x has the same resolution as the sample natural image Image 1x. Then, the first down-sampled image Image 2x can be subjected to repeated processing of the above steps to obtain a second down-sampled image Image 4x and a second residual image Res 2x corresponding to the first down-sampled image Image 2x. The second residual image Res 2x has the same resolution as the first down-sampled image Image 2x. Then, the second down-sampled image Image 4x, the first residual image Res 1x, and the second residual image Res 2x can be integrated to obtain a Laplacian image pyramid corresponding to the sample natural image. More levels of down-sampled images and residual images can be obtained in the same manner as described above, which will not be described here.

[0103] In some possible implementation manners, if each generator includes a plurality of to-be-trained sub-level generators, the images in the first multi-scale image can be respectively input into the plurality of to-be-trained sub-level generators to obtain multi-scale sub-generated stylized images corresponding to the first multi-scale image. Figure 4is a schematic view of determining a multi-scale sub-generated stylized image corresponding to the first multi-scale image provided by an embodiment of the present application. In actual application, if the first generator includes 3 to-be-trained sub-level generators, i.e., a first to-be-trained sub-level generator G_1, a second to-be-trained sub-level generator G_2 and a third to-be-trained sub-level generator G_3, in a single training process, the second down-sampled image Image 4x can be input into the first to-be-trained sub-level generator G_1, and a first scale sub-generated stylized image 4x is output. Then the first scale sub-generated stylized image 4x can be up-sampled, and the up-sampled image is input into the second to-be-trained sub-level generator G_2 after being spliced with the second residual image Res 2x, and a second scale sub-generated stylized image 2x is output. Then, the second scale sub-generated stylized image 2x can be up-sampled, and the up-sampled image is input into the third to-be-trained sub-level generator G_3 after being spliced with the first residual image Res 1x, and a third scale sub-generated stylized image 1x is output.

[0104] S205: inputting the sample stylized image into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and inputting the generated natural image into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image.

[0105] In the embodiment of the present application, after obtaining the sample stylized image, a Laplacian image pyramid corresponding to the sample stylized image can be constructed as the overall input of the generator. The specific construction steps are consistent with the construction of the Laplacian image pyramid corresponding to the sample natural image in the above, which will not be repeated here.

[0106] In some possible implementations, in a single training process, the sample stylized image B can be first input into the second to-be-trained generator G_B, and a generated paper-cut image Fake A is output. Then the generated paper-cut image Fake A can be input into the first to-be-trained generator G_A, and a reconstructed paper-cut image Rec B is output. The sample stylized image B, the generated paper-cut image Fake A and the reconstructed paper-cut image Rec B can be images containing the same type of objects. For example, the sample stylized image B, the generated paper-cut image Fake A and the reconstructed paper-cut image Rec B can be images containing buildings.

[0107] In some possible implementations, if each generator includes a plurality of to-be-trained sub-level generators, the Laplacian image pyramid corresponding to the sample stylized image can be input into the plurality of to-be-trained sub-level generators respectively, to obtain a multi-scale sub-generated stylized image corresponding to the Laplacian image pyramid corresponding to the sample stylized image.

[0108] S207: constructing the target loss function according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image, and the reconstructed stylized image.

[0109] In the embodiments of the present application, the sample natural image, the generated stylized image, and the reconstructed natural image can be respectively input into an edge extractor for edge feature extraction processing to obtain the first edge feature corresponding to the sample natural image, the second edge feature corresponding to the generated stylized image, and the third edge feature corresponding to the reconstructed natural image. Then, the first loss function can be constructed according to the first edge feature, the second edge feature, and the third edge feature, the second loss function can be constructed according to the difference information of the sample stylized image and the reconstructed stylized image, and the target loss function can be constructed according to the first loss function and the second loss function. Optionally, the edge extractor can adopt the structure of a Holistically-Nested Edge Detection (HED) network, and the sample natural image A, the generated stylized image Fake B, and the reconstructed natural image Rec A are respectively input into the edge extractor to obtain the first edge feature EDGE(A) corresponding to the sample natural image A, the second edge feature EDGE(Fake B) corresponding to the generated stylized image Fake B, and the third edge feature EDGE(Rec A) corresponding to the reconstructed natural image Rec A. The edge extractor can also adopt the structure of an Edge detection (DeeoEdge) network, and can also adopt the structure of a Deep Category-aware Semantic Edge Detection (CaseNet) network, which is not limited in the embodiments of the present application. By extracting the edge features of the sample natural image, the generated stylized image, and the reconstructed natural image, the information flow difference from the natural image to the paper-cut image and from the paper-cut image to the natural image can be balanced.

[0110] In some possible implementations, the first loss value can be determined according to the difference between the first edge feature and the second edge feature, and the second sub-loss value can be determined according to the difference between the first edge feature and the third edge feature. Then, the first loss value can be determined according to the sum of the first sub-loss value and the second sub-loss value. Optionally, the L1 Loss or L2 Loss between the first edge feature EDGE(A) corresponding to the sample natural image A and the second edge feature EDGE(Fake B) corresponding to the generated stylized image Fake B can be determined, and the L1 Loss or L2 Loss between the first edge feature EDGE(A) corresponding to the sample natural image A and the third edge feature EDGE(Rec A) corresponding to the reconstructed natural image Rec A can be determined. Then, the first loss function Loss1 can be obtained by weighted sum of the two.

[0111] In the embodiments of the present application, the L1 Loss or L2 Loss between the sample stylized image and the reconstructed stylized image can be determined as the second loss function Loss2. Optionally, the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image can all be represented by matrices. In actual applications, the second loss function can be determined according to the difference between the corresponding values in the matrix corresponding to the sample stylized image and the matrix corresponding to the reconstructed stylized image.

[0112] In the embodiments of the present application, after obtaining the first loss function and the second loss function, the target loss function can be constructed according to the sum of the first loss function and the second loss function.

[0113] In some possible implementations, the generated stylized image and the sample stylized image can be input into the first discriminator to obtain the first discrimination value corresponding to the generated stylized image and the second discrimination value corresponding to the sample stylized image. Then, the third loss function can be determined according to the sum of the difference between the first label value corresponding to the generated stylized image and the first discrimination value and the difference between the second label value corresponding to the sample stylized image and the second discrimination value. In the second stage of the adversarial learning to determine whether the image is a paper-cut image, the first generator G_A can be optimized, and the first discriminator D_A can not be updated. The first label value label corresponding to the generated stylized image Fake B can be set to 1 (true) in advance, and the second label value label corresponding to the sample stylized image B can be set to 0 (false). Then, the generated stylized image Fake B and the sample stylized image B can be input into the first discriminator D_A to obtain the respective reduced dimension features, and the third loss function can be determined according to the respective reduced dimension features and the respective labels. The third loss function can also adopt L2 Loss or cross-entropy Loss.

[0114] In some possible implementation manners, the generated natural image and the sample natural image can be input into the second discriminator to obtain a third discrimination value corresponding to the generated natural image and a fourth discrimination value corresponding to the sample natural image. Then, a fourth loss function can be determined according to a sum of a difference between the first label value corresponding to the generated natural image and the third discrimination value and a difference between the second label value corresponding to the sample natural image and the fourth discrimination value. In the second stage of the adversarial learning of judging whether the image is a natural image, the second generator G_B can be optimized, and the second discriminator D_A is not updated. The first label value label corresponding to the generated natural image Fake A can be set as 1 (true) in advance, and the second label value label corresponding to the sample natural image A can be set as 0 (false). Then, the generated natural image Fake A and the sample natural image A can be input into the second discriminator D_B to obtain respective reduced dimension features, and the fourth loss function can be determined according to the respective reduced dimension features and the respective labels. The fourth loss function can also adopt L2 Loss or cross-entropy Loss.

[0115] In some possible implementation manners, after the first loss function, the second loss function, the third loss function and the fourth loss function are obtained, a target loss function can be constructed according to a sum of the first loss function, the second loss function, the third loss function and the fourth loss function.

[0116] In the embodiments of the present application, before the generated stylized image and the sample stylized image are input into the first discriminator to obtain the first discrimination value corresponding to the generated stylized image and the second discrimination value corresponding to the sample stylized image, the first discriminator can be trained. Optionally, the generated stylized image and the sample stylized image can be input into the first to-be-trained discriminator to obtain the first candidate discrimination value corresponding to the generated stylized image and the second candidate discrimination value corresponding to the sample stylized image. Then, the fifth loss function can be determined according to the sum of the difference between the second label value corresponding to the generated stylized image and the first candidate discrimination value and the difference between the first label value corresponding to the sample stylized image and the second candidate discrimination value. Further, the model parameters of the first to-be-trained discriminator can be adjusted based on the fifth loss function until the second preset training end condition is met to end the training, and the first discriminator is obtained. That is, in the first stage of the adversarial learning of judging whether the image is a paper-cut image, the first to-be-trained discriminator can be optimized to obtain the first discriminator, and the first generator D_A is not updated. The second label value label corresponding to the generated stylized image Fake B can be preset as 0 (false), and the first label value label corresponding to the sample stylized image B can be preset as 1 (true). Then, the generated stylized image Fake B and the sample stylized image B can be input into the first to-be-trained discriminator to obtain respective reduced dimension features, and the fifth loss function can be determined according to the respective reduced dimension features and the respective labels. The fifth loss function can also use L2Loss or cross-entropy Loss. Further, the model parameters of the first to-be-trained discriminator can be adjusted based on the fifth loss function until the second preset training end condition is met to end the training, and the first discriminator is obtained. The second preset training end condition can be that the value corresponding to the fifth loss function is less than a preset threshold.

[0117] In the embodiments of the present application, before the generated natural image and the sample natural image are input into the second discriminator to obtain the third discrimination value corresponding to the generated natural image and the fourth discrimination value corresponding to the sample natural image, the second discriminator can be trained. Alternatively, the generated natural image and the sample natural image can be input into the second to-be-trained discriminator to obtain the third candidate discrimination value corresponding to the generated natural image and the fourth candidate discrimination value corresponding to the sample natural image. Then, the sixth loss function can be determined according to the sum of the difference between the second label value corresponding to the generated natural image and the third candidate discrimination value and the difference between the first label value corresponding to the sample natural image and the fourth candidate discrimination value. Further, the model parameters of the second to-be-trained discriminator can be adjusted based on the sixth loss function until the third preset training end condition is met to end the training and obtain the second discriminator. That is, in the first stage of the adversarial learning of judging whether the image is a natural image, the second to-be-trained discriminator can be optimized to obtain the second discriminator, and the second generator is not updated. The second label value corresponding to the generated natural image Fake A can be pre-set as 0 (false), and the first label value corresponding to the sample natural image A can be pre-set as 1 (true). Then, the generated natural image Fake A and the sample natural image A can be input into the second to-be-trained discriminator to obtain the respective reduced dimension features, and the sixth loss function can be determined according to the respective reduced dimension features and the respective labels. The sixth loss function can also use L2 Loss or cross-entropy Loss. Further, the model parameters of the second to-be-trained discriminator can be adjusted based on the sixth loss function until the third preset training end condition is met to end the training and obtain the second discriminator. The third preset training end condition can be that the value corresponding to the fifth loss function is less than a preset threshold.

[0118] In some possible implementation manners, if each generator comprises a plurality of to-be-trained sub-level generators, a first multi-scale image corresponding to a sample natural image and a multi-scale sub-generated stylized image corresponding to the first multi-scale image can be obtained. The first multi-scale image can be equal in number to the number of levels of the to-be-trained sub-level generators. Then, the first multi-scale image, the multi-scale sub-generated stylized image, and the reconstructed image can be input into an edge extractor for edge feature extraction processing, to obtain a first multi-scale edge feature corresponding to the first multi-scale image, a second multi-scale edge feature corresponding to the multi-scale sub-generated stylized image, and a third edge feature corresponding to the reconstructed natural image. Then, a first loss function can be constructed according to the first multi-scale edge feature, the second multi-scale edge feature, and the third edge feature. The specific construction manner is the same as that described above, and will not be described here again. Further, a second multi-scale image corresponding to the sample natural image can be obtained by performing down-sampling processing on the sample natural image, and the second multi-scale image can be equal in number to the number of levels of the to-be-trained sub-level generators. Then, the first multi-scale image and the second multi-scale image can be input into a first discriminator to obtain a first reference discrimination value corresponding to each image in the first multi-scale image and a second reference discrimination value corresponding to each image in the second multi-scale image, and a third loss function can be determined according to a sum of a difference between a first label value corresponding to each image in the first multi-scale image and the first reference discrimination value and a difference between a second label value corresponding to each image in the second multi-scale image and the second reference discrimination value. The specific determination manner is the same as that described above, and will not be described here again. The optimization objective is constructed by mutual conversion between the natural image and the stylized image, which can improve the robustness of the image generation model and improve the quality and resolution of the generated stylized image.

[0119] S209: training the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training; wherein the first to-be-trained generator at the end of the training is used as the image generation model.

[0120] In the embodiments of the present application, after the target loss function is constructed, the parameters of the first generator and the second generator can be adjusted based on the target loss function until a first preset training end condition is met to end the training, and the first to-be-trained generator at the end of the training can be used as the image generation model. The first preset training end condition can be that the value corresponding to the target loss function is less than a preset threshold.

[0121] The training method of the image generation model provided in the embodiments of the present application can improve the robustness of the image generation model and improve the quality and resolution of the generated stylized image by constructing an optimization target through mutual conversion of natural images and stylized images. By using a multi-level generator to gradually generate and modify the results of the previous level, a high-quality, high-resolution paper-cut image can be further generated. By constructing a Laplace image pyramid, the information of the image can be enriched, and a high-quality, high-resolution paper-cut image can be further generated. By extracting the edge features of the sample natural image, the generated stylized image and the reconstructed natural image, the information flow difference from the natural image to the paper-cut image and from the paper-cut image to the natural image can be balanced.

[0122] The embodiments of the present application also provide a training device of an image generation model, Figure 5 is a structural schematic diagram of an image generation device provided by the embodiments of the present application, as Figure 5 indicated, the image generation device can include:

[0123] The first acquisition module 501 is configured to acquire a sample natural image and a sample stylized image.

[0124] The first processing module 503 is configured to input the sample natural image into a first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and input the generated stylized image into a second to-be-trained generator for natural generation processing to obtain a reconstructed natural image.

[0125] The second processing module 505 is configured to input the sample stylized image into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and input the generated natural image into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image.

[0126] The construction module 507 is configured to construct a target loss function according to the sample natural image, the generated stylized image, the reconstructed natural image, the sample stylized image, the generated natural image and the reconstructed stylized image.

[0127] The first training module 509 is configured to train the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training.

[0128] The first to-be-trained generator at the end of the training is used as the image generation model.

[0129] In some possible implementation manners, the construction module includes:

[0130] The edge feature extraction submodule is configured to input the sample natural image, the generated stylized image and the reconstructed natural image into an edge extractor respectively for edge feature extraction processing to obtain a first edge feature corresponding to the sample natural image, a second edge feature corresponding to the generated stylized image and a third edge feature corresponding to the reconstructed natural image.

[0131] The first construction submodule is configured to construct a first loss function according to the first edge feature, the second edge feature and the third edge feature.

[0132] The second construction submodule is configured to construct a second loss function according to difference information of the sample stylized image and the reconstructed stylized image.

[0133] The third construction submodule is configured to construct a target loss function according to the first loss function and the second loss function.

[0134] In some possible implementation manners, the first construction submodule is configured to determine a first sub-loss value according to a difference between the first edge feature and the second edge feature.

[0135] determine a second sub-loss value according to a difference between the first edge feature and the third edge feature.

[0136] determine the first loss function according to a sum of the first sub-loss value and the second sub-loss value.

[0137] On the other hand, the third construction submodule includes:

[0138] The first input-output unit is configured to input the generated stylized image and the sample stylized image into the first discriminator to obtain a first discrimination value corresponding to the generated stylized image and a second discrimination value corresponding to the sample stylized image.

[0139] The first determination unit is configured to determine a third loss function according to a sum of a difference between a first label value corresponding to the generated stylized image and the first discrimination value and a difference between a second label value corresponding to the sample stylized image and the second discrimination value.

[0140] The second input-output unit is configured to input the generated natural image and the sample natural image into the second discriminator to obtain a third discrimination value corresponding to the generated natural image and a fourth discrimination value corresponding to the sample natural image.

[0141] The second determination unit is configured to determine a fourth loss function according to a sum of a difference between a first label value corresponding to the generated natural image and the third discrimination value and a difference between a second label value corresponding to the sample natural image and the fourth discrimination value.

[0142] The construction unit is configured to construct a target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function.

[0143] In some possible implementation manners, the training apparatus of the image generation model further includes a second training module;

[0144] The second training module is configured to train the first discriminator and the second discriminator.

[0145] The training of the first discriminator includes:

[0146] The generated stylized image and the sample stylized image are input into the first to-be-trained discriminator to obtain a first candidate discrimination value corresponding to the generated stylized image and a second candidate discrimination value corresponding to the sample stylized image.

[0147] A fifth loss function is determined according to a sum value of a difference between the second label value corresponding to the generated stylized image and the first candidate discrimination value and a difference between the first label value corresponding to the sample stylized image and the second candidate discrimination value.

[0148] The model parameters of the first to-be-trained discriminator are adjusted based on the fifth loss function until a second preset training end condition is met to end the training, and the first discriminator is obtained.

[0149] The training of the second discriminator includes:

[0150] The generated natural image and the sample natural image are input into the second to-be-trained discriminator to obtain a third candidate discrimination value corresponding to the generated natural image and a fourth candidate discrimination value corresponding to the sample natural image.

[0151] A sixth loss function is determined according to a sum value of a difference between the second label value corresponding to the generated natural image and the third candidate discrimination value and a difference between the first label value corresponding to the sample natural image and the fourth candidate discrimination value.

[0152] The model parameters of the second to-be-trained discriminator are adjusted based on the sixth loss function until a third preset training end condition is met to end the training, and the second discriminator is obtained.

[0153] In some possible implementation manners, the first to-be-trained generator includes a plurality of to-be-trained sub-level generators.

[0154] The edge feature extraction submodule is configured to obtain a first multi-scale image corresponding to the sample natural image and a multi-scale sub-generated stylized image corresponding to the first multi-scale image; the scale of the first multi-scale image is equal to the number of levels of the to-be-trained sub-level generator.

[0155] The first multi-scale image, the multi-scale sub-generated stylized image and the reconstructed natural image are input into the edge extractor for edge feature extraction processing to obtain a first multi-scale edge feature corresponding to the first multi-scale image, a second multi-scale edge feature corresponding to the multi-scale sub-generated stylized image and a third edge feature corresponding to the reconstructed natural image.

[0156] The first construction submodule is configured to construct a first loss function according to the first multi-scale edge feature, the second multi-scale edge feature, and the third edge feature.

[0157] In some possible implementation manners, the first input / output unit is configured to acquire a second multi-scale image corresponding to the sample natural image; the second multi-scale image is an image obtained by performing down-sampling processing on the sample natural image;

[0158] The first multi-scale image and the second multi-scale image are input into the first discriminator to obtain a first reference discrimination value corresponding to each image in the first multi-scale image and a second reference discrimination value corresponding to each image in the second multi-scale image;

[0159] The first determination unit is configured to determine a third loss function according to a difference value between the first label value corresponding to each image in the first multi-scale image and the first reference discrimination value and a sum value of a difference between the second label value corresponding to each image in the second multi-scale image and the second reference discrimination value.

[0160] In some possible implementation manners, the edge feature extraction submodule is configured to perform down-sampling processing on the sample natural image to obtain a down-sampled image corresponding to the current sampling processing;

[0161] The down-sampled image is subjected to up-sampling processing to obtain an up-sampled image corresponding to the current sampling processing; the up-sampled image and the sample natural image have the same resolution;

[0162] According to difference information between the sample natural image and the up-sampled image, a residual image corresponding to the current sampling processing is determined;

[0163] The down-sampled image corresponding to the current sampling processing is taken as the sample natural image to perform sampling processing, to obtain a residual image set corresponding to the multi-time sampling processing;

[0164] According to the down-sampled image corresponding to the last sampling processing in the multi-time sampling processing and the residual image set corresponding to the multi-time sampling processing, the first multi-scale image is constructed.

[0165] The images in the first multi-scale image are input into the plurality of to-be-trained sub-level generators respectively, to obtain multi-scale sub-generated stylized images corresponding to the first multi-scale image.

[0166] The device and the method embodiment in the embodiment of the application are based on the same application concept.

[0167] The following introduces a specific embodiment of an image generation method of the application, Figure 6 is a flowchart of an image generation method provided by the embodiment of the application, Figure 7is a schematic diagram of an image generation method provided by an embodiment of the present application. The present specification provides method operation steps as shown in the embodiments or flowcharts, but more or fewer operation steps can be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is only one of many execution orders, and does not represent the only execution order. In actual execution, the method order shown in the embodiments or the accompanying drawings can be executed in sequence or in parallel (for example, in a parallel processor or multi-thread processing environment).

[0168] As shown in particular in Figure 6 and 7 , the image generation method can include:

[0169] S601: Obtain a natural image to be processed.

[0170] In the embodiments of the present application, an image obtained by a camera shooting objects such as scenes, objects, and people in the physical world can be obtained as the natural image to be processed.

[0171] S603: Input the natural image to be processed into an image generation model, and output a target stylized image corresponding to the natural image to be processed.

[0172] In the embodiments of the present application, the image generation model can be an image generation model obtained based on the training method of the image generation model described above.

[0173] In the application stage, the second generator, the first discriminator, and the second discriminator can be discarded, and only the natural image to be processed is required to construct a Laplacian image pyramid and input into the first generator for stylized generation processing to obtain a target stylized image, i.e., a target paper-cut image. By converting the natural image into a paper-cut image, an artistic creation platform can be provided to users as a small program during festivals such as the Chinese New Year, improving user experience and promoting social sharing.

[0174] In some possible implementations, after obtaining the target stylized image, post-processing can be performed on the target stylized image to obtain a final result. Optionally, the target stylized image can be subjected to a binarization process, then connected domains are detected, and connected domains with an area less than a certain proportion (such as 2%) are filled with the color around them. The black-and-white binary image can also be converted into a red-and-white binary image, such as mapping black to red and keeping white unchanged, to highlight the festive atmosphere.

[0175] By using the image generation method provided by the embodiments of the present application, the quality and resolution of the generated stylized image can be improved.

[0176] An image generation apparatus provided by an embodiment of the present application, Figure 8 is a structural schematic diagram of an image generation apparatus provided by an embodiment of the present application, as shown inFigure 8 As shown in the figure, the image generation apparatus can comprise:

[0177] The second acquisition module 801 is configured to acquire a natural image to be processed.

[0178] The input and output module 803 is configured to input the natural image to be processed into the image generation model and output a target stylized image corresponding to the natural image to be processed.

[0179] The device and method embodiments in the present application are based on the same application concept.

[0180] The electronic device provided in the embodiments of the present application comprises a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the training method of the image generation model or the image generation method provided in the above method embodiments.

[0181] Figure 9 is a hardware structure schematic diagram of an electronic device for implementing the training method of the image generation model or the image generation method provided in the embodiments of the present application. The electronic device can participate in constituting or containing the training apparatus of the image generation model or the image generation apparatus provided in the embodiments of the present application. As shown in the figure, Figure 9 The electronic device can comprise one or more (in the figure, processors 901a and 901b are used to show) processors 901 (the processor 901 can comprise but is not limited to a microprocessor 901 MCU or a programmable logic device FPGA and other processing devices), a memory 903 for storing data, and a transmission device 905 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface and / or a power supply. Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also comprise more or less components than those shown in Figure 9 or have a different configuration from that shown in Figure 9

[0182] ​It should be noted that the one or more processors 901 and / or other data processing circuitry described above can be referred to as "data processing circuitry" in the present application. The data processing circuitry can be embodied in whole or in part as software, hardware, firmware, or any combination thereof. In addition, the data processing circuitry can be a single independent processing module, or be incorporated into any one of other elements in the electronic device (or mobile device) in whole or in part. As referred to in the embodiments of the present application, the data processing circuitry as a processor 901 controls (for example, the selection of the variable resistance terminal path connected with the interface).

[0183] The memory 903 can be used to store software programs and modules of application software, such as program instructions / data storage means corresponding to the training method of an image generation model or the image generation method in the embodiments of the present application. The processor 901 implements the above-mentioned training method of an image generation model or image generation method by running the software programs and modules stored in the memory 903 to perform various functional applications and data processing. The memory 903 can include a high-speed random access memory, and can also include a non-volatile random access memory 903, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories 903. In some possible embodiments, the memory 903 can further include a memory 903 disposed remotely with respect to the processor, and these remote memories 903 can be connected to the electronic device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The transmission device 905 is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the electronic device. In one example, the transmission device 905 includes a network interface controller (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 905 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0185] The display can be, for example, a liquid crystal display (LCD) in the form of a touch screen, which can enable a user to interact with the user interface of the electronic device (or mobile device).

[0186] The embodiment of the present application provides a computer readable storage medium, which can be arranged in an electronic device to save at least one instruction or at least one program related to the training method of the image generation model or the image generation method in the method embodiment, and the at least one instruction or the at least one program is loaded and executed by the processor to realize the training method of the image generation model or the image generation method provided by the above method embodiment.

[0187] Optionally, in the embodiment, the storage medium can be located in at least one of a plurality of network servers of a computer network. Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing program codes.

[0188] It should be noted that the above-mentioned order of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments, and the above-mentioned description is for specific embodiments, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in different orders in different embodiments and can achieve the expected results. In addition, the processes depicted in the drawings do not necessarily require a specific order or connection order to achieve the desired results, and in some embodiments, multi-task parallel processing is possible or can be advantageous.

[0189] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment mainly describes the differences from other embodiments. Especially, the embodiments of the device and the electronic device are described simply because they are based on the similar method embodiments, and the related parts can be referred to the part of the method embodiments.

[0190] The above is the preferred embodiment of the present application, and it should be noted that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements are also considered to be within the scope of protection of the present application.

Claims

1. A method for training an image generation model, characterized in that, The method comprises: obtaining a sample natural image and a sample stylized image; inputting the sample natural image into a first to-be-trained generator for stylized generation processing to obtain a generated stylized image, and inputting the generated stylized image into a second to-be-trained generator for natural generation processing to obtain a reconstructed natural image; inputting the sample stylized image into the second to-be-trained generator for natural generation processing to obtain a generated natural image, and inputting the generated natural image into the first to-be-trained generator for stylized generation processing to obtain a reconstructed stylized image; the first to-be-trained generator comprises a plurality of to-be-trained sub-level generators; obtaining a first multi-scale image corresponding to the sample natural image and a multi-scale sub-generated stylized image corresponding to the first multi-scale image; the scale of the first multi-scale image is equal to the number of levels of the to-be-trained sub-level generators; inputting the first multi-scale image, the multi-scale sub-generated stylized image and the reconstructed natural image into an edge extractor respectively for edge feature extraction processing to obtain a first multi-scale edge feature corresponding to the first multi-scale image, a second multi-scale edge feature corresponding to the multi-scale sub-generated stylized image and a third edge feature corresponding to the reconstructed natural image; the parameters of the edge extractor remain unchanged throughout the training process; constructing a first loss function according to the first multi-scale edge feature, the second multi-scale edge feature and the third edge feature; constructing a second loss function according to the difference information of the sample stylized image and the reconstructed stylized image; constructing a target loss function according to the first loss function and the second loss function; training the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training; wherein the first to-be-trained generator at the end of the training is used as the image generation model.

2. The method of claim 1, wherein, The method comprises: determining a first sub-loss value according to the difference between the first multi-scale edge feature and the second multi-scale edge feature; determining a second sub-loss value according to the difference between the first multi-scale edge feature and the third edge feature; determining the first loss function according to the sum of the first sub-loss value and the second sub-loss value.

3. The method of claim 1, wherein, The method comprises: inputting the generated stylized image and the sample stylized image into a first discriminator to obtain a first discrimination value corresponding to the generated stylized image and a second discrimination value corresponding to the sample stylized image; determining a third loss function according to the sum of the difference between a first label value corresponding to the generated stylized image and the first discrimination value and the difference between a second label value corresponding to the sample stylized image and the second discrimination value; inputting the generated natural image and the sample natural image into a second discriminator to obtain a third discrimination value corresponding to the generated natural image and a fourth discrimination value corresponding to the sample natural image; determining a fourth loss function according to a sum value of a difference between the first label value corresponding to the generated natural image and the third discrimination value and a difference between the second label value corresponding to the sample natural image and the fourth discrimination value; constructing the target loss function according to the first loss function, the second loss function, the third loss function and the fourth loss function.

4. The method of claim 3, wherein, Further comprising: training the first discriminator and the second discriminator; the training of the first discriminator comprises: inputting the generated stylized image and the sample stylized image into a first to-be-trained discriminator to obtain a first candidate discrimination value corresponding to the generated stylized image and a second candidate discrimination value corresponding to the sample stylized image; determining a fifth loss function according to a sum value of a difference between the second label value corresponding to the generated stylized image and the first candidate discrimination value and a difference between the first label value corresponding to the sample stylized image and the second candidate discrimination value; adjusting model parameters of the first to-be-trained discriminator based on the fifth loss function until a second preset training end condition is met to end the training, and obtaining the first discriminator; the training of the second discriminator comprises: inputting the generated natural image and the sample natural image into a second to-be-trained discriminator to obtain a third candidate discrimination value corresponding to the generated natural image and a fourth candidate discrimination value corresponding to the sample natural image; determining a sixth loss function according to a sum value of a difference between the second label value corresponding to the generated natural image and the third candidate discrimination value and a difference between the first label value corresponding to the sample natural image and the fourth candidate discrimination value; adjusting model parameters of the second to-be-trained discriminator based on the sixth loss function until a third preset training end condition is met to end the training, and obtaining the second discriminator.

5. The method of claim 3, wherein, the inputting of the generated stylized image and the sample stylized image into the first discriminator to obtain the first discrimination value corresponding to the generated stylized image and the second discrimination value corresponding to the sample stylized image comprises: obtaining a second multi-scale image corresponding to the sample natural image; the second multi-scale image is an image obtained by performing down-sampling processing on the sample natural image; inputting the first multi-scale image and the second multi-scale image into the first discriminator to obtain a first reference discrimination value corresponding to each image in the first multi-scale image and a second reference discrimination value corresponding to each image in the second multi-scale image; the determining of the third loss function according to the sum value of the difference between the first label value corresponding to the generated stylized image and the first discrimination value and the difference between the second label value corresponding to the sample stylized image and the second discrimination value comprises: Determine the third loss function according to the difference value of the first reference discriminant value and the first annotation value corresponding to each image in the first multi-scale image and the sum value of the difference value of the second reference discriminant value and the second annotation value corresponding to each image in the second multi-scale image.

6. The method of claim 3, wherein, The obtaining of the first multi-scale image corresponding to the sample natural image and the multi-scale sub-generated stylized image corresponding to the first multi-scale image comprises: performing down-sampling processing on the sample natural image to obtain a down-sampled image corresponding to the down-sampling processing; performing up-sampling processing on the down-sampled image to obtain an up-sampled image corresponding to the down-sampling processing; the resolution of the up-sampled image is the same as that of the sample natural image; determining a residual image corresponding to the down-sampling processing according to the difference information between the sample natural image and the up-sampled image; performing sampling processing on the down-sampled image corresponding to the down-sampling processing as a sample natural image to obtain a residual image set corresponding to multiple sampling processing; constructing the first multi-scale image according to the down-sampled image corresponding to the last sampling processing in the multiple sampling processing and the residual image set corresponding to the multiple sampling processing; inputting the images in the first multi-scale image into the multiple to-be-trained sub-level generators respectively to obtain the multi-scale sub-generated stylized image corresponding to the first multi-scale image.

7. An image generation method characterized by, comprise: obtaining a natural image to be processed; inputting the natural image to be processed into an image generation model to output a target stylized image corresponding to the natural image to be processed; the image generation model is obtained based on the training method of any one of claims 1-6.

8. An apparatus for training an image generation model, comprising: comprise: a first acquisition module configured to acquire a sample natural image and a sample stylized image; a first processing module configured to input the sample natural image into a first to-be-trained generator to perform stylized generation processing, obtain a generated stylized image, and input the generated stylized image into a second to-be-trained generator to perform natural generation processing, and obtain a reconstructed natural image; a second processing module configured to input the sample stylized image into the second to-be-trained generator to perform natural generation processing, obtain a generated natural image, and input the generated natural image into the first to-be-trained generator to perform stylized generation processing, and obtain a reconstructed stylized image; the first to-be-trained generator comprises multiple to-be-trained sub-level generators; an edge feature extraction submodule configured to acquire a first multi-scale image corresponding to the sample natural image and a multi-scale sub-generated stylized image corresponding to the first multi-scale image; the scale of the first multi-scale image is equal to the number of levels of the to-be-trained sub-level generator; input the first multi-scale image, the multi-scale sub-generated stylized image and the reconstructed natural image into an edge extractor for edge feature extraction processing to obtain a first multi-scale edge feature corresponding to the first multi-scale image, a second multi-scale edge feature corresponding to the multi-scale sub-generated stylized image, and a third edge feature corresponding to the reconstructed natural image; the parameters of the edge extractor remain unchanged throughout the training process; The first constructing submodule is configured to construct a first loss function according to the first multi-scale edge feature, the second multi-scale edge feature, and the third edge feature. The second constructing submodule is configured to construct a second loss function according to difference information of the sample stylized image and the reconstructed stylized image. The third constructing submodule is configured to construct a target loss function according to the first loss function and the second loss function. The first training module is configured to train the first to-be-trained generator and the second to-be-trained generator based on the target loss function until a first preset training end condition is met to end the training. The first to-be-trained generator at the end of the training is taken as the image generation model.

9. The apparatus of claim 8, wherein, The first constructing submodule is configured to: determine a first sub-loss value according to a difference between the first multi-scale edge feature and the second multi-scale edge feature; determine a second sub-loss value according to a difference between the first multi-scale edge feature and the third edge feature; and determine the first loss function according to a sum of the first sub-loss value and the second sub-loss value.

10. The apparatus of claim 8, wherein, The third constructing submodule includes: The first input-output unit is configured to input the generated stylized image and the sample stylized image into a first discriminator to obtain a first discrimination value corresponding to the generated stylized image and a second discrimination value corresponding to the sample stylized image. The first determining unit is configured to determine a third loss function according to a sum of a difference between a first label value corresponding to the generated stylized image and the first discrimination value and a difference between a second label value corresponding to the sample stylized image and the second discrimination value. The second input-output unit is configured to input the generated natural image and the sample natural image into a second discriminator to obtain a third discrimination value corresponding to the generated natural image and a fourth discrimination value corresponding to the sample natural image. The second determining unit is configured to determine a fourth loss function according to a sum of a difference between the first label value corresponding to the generated natural image and the third discrimination value and a difference between the second label value corresponding to the sample natural image and the fourth discrimination value. The constructing unit is configured to construct the target loss function according to the first loss function, the second loss function, the third loss function, and the fourth loss function.

11. The apparatus of claim 10, wherein, The training device of the image generation model further includes a second training module. The second training module is configured to train the first discriminator and the second discriminator. The training of the first discriminator includes: inputting the generated stylized image and the sample stylized image into a first to-be-trained discriminator to obtain a first candidate discrimination value corresponding to the generated stylized image and a second candidate discrimination value corresponding to the sample stylized image; determining a fifth loss function according to a sum of a difference between the second label value corresponding to the generated stylized image and the first candidate discrimination value and a difference between the first label value corresponding to the sample stylized image and the second candidate discrimination value. adjusting model parameters of the first to-be-trained discriminator based on the fifth loss function until a second preset training end condition is met to end the training, and obtaining the first discriminator; the training of the second discriminator comprises: inputting the generated natural image and the sample natural image into a second to-be-trained discriminator to obtain a third candidate discriminant value corresponding to the generated natural image and a fourth candidate discriminant value corresponding to the sample natural image; determining a sixth loss function according to a sum value of a difference value of the second label value corresponding to the generated natural image and the third candidate discriminant value and a difference value of the first label value corresponding to the sample natural image and the fourth candidate discriminant value; adjusting model parameters of the second to-be-trained discriminator based on the sixth loss function until a third preset training end condition is met to end the training, and obtaining the second discriminator.

12. The apparatus of claim 10, wherein, The first input and output unit is configured to obtain a second multi-scale image corresponding to the sample natural image; the second multi-scale image is an image obtained by performing down-sampling processing on the sample natural image; inputting the first multi-scale image and the second multi-scale image into the first discriminator to obtain a first reference discriminant value corresponding to each image in the first multi-scale image and a second reference discriminant value corresponding to each image in the second multi-scale image; The first determination unit is configured to determine the third loss function according to a sum value of a difference value of the first label value corresponding to each image in the first multi-scale image and the first reference discriminant value and a difference value of the second label value corresponding to each image in the second multi-scale image and the second reference discriminant value.

13. The apparatus of claim 10, wherein, The edge feature extraction sub-module is configured to: performing down-sampling processing on the sample natural image to obtain a down-sampled image corresponding to the current sampling processing; performing up-sampling processing on the down-sampled image to obtain an up-sampled image corresponding to the current sampling processing; the up-sampled image and the sample natural image have the same resolution; determining a residual image corresponding to the current sampling processing according to difference information of the sample natural image and the up-sampled image; performing sampling processing on the down-sampled image corresponding to the current sampling processing as a sample natural image to obtain a residual image set corresponding to a plurality of sampling processing; constructing the first multi-scale image according to a down-sampled image corresponding to the last sampling processing in the plurality of sampling processing and the residual image set corresponding to the plurality of sampling processing; inputting images in the first multi-scale image into the plurality of to-be-trained sub-level generators respectively to obtain multi-scale sub-generated stylized images corresponding to the first multi-scale image.

14. An image generation apparatus characterized by comprising: comprises: a second acquisition module configured to acquire a natural image to be processed; an input and output module configured to input the natural image to be processed into an image generation model and output a target stylized image corresponding to the natural image to be processed; the image generation model is a model obtained based on the training method of any one of claims 1-6.

15. An electronic device, comprising: The electronic device includes a processor and a memory, the memory having stored therein at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training method of the image generation model according to any one of claims 1-6, or to implement the image generation method according to claim 7.

16. A computer storage medium, comprising, The storage medium has stored therein at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training method of the image generation model according to any one of claims 1-6, or to implement the image generation method according to claim 7.

17. A computer program product, characterised in that, The computer program product includes at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training method of the image generation model according to any one of claims 1-6, or to implement the image generation method according to claim 7.

Images