Image processing method and storage medium

Abstract

The disclosed embodiment discloses an image processing method and storage medium, wherein the method comprises: training a generative adversarial network using real-life images and cartoon images; wherein the generative adversarial network comprises a generative network and a discriminative network; receiving an image to be converted; processing the image to be converted based on the generative network to obtain a cartoonized target image; wherein the generative network is obtained based on unsupervised training combined with the discriminative network; and utilizing a deep learning method to realize cartoonization of real-life images in an end-to-end manner, with higher image conversion efficiency.

Description

Technical Field

[0001] The present disclosure relates to computer vision technology, and in particular to an image processing method and a storage medium. Background Art

[0002] Many people are attracted by the scenery in cartoons, and there are also related researches devoted to making the real scenery have a style similar to cartoons. Therefore, related technologies for cartoonizing real-life images have emerged. Cartoonizing real-life images is originally a very difficult task. The existing methods often cause a certain degree of deformation and distortion in the main content of the image after cartoonization, and the color will also be blurred in a small range, making it difficult to ensure a good visual cartoon effect. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide an image processing method and a storage medium.

[0004] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0005] Using real-life images and cartoon images to train a generative adversarial network; wherein the generative adversarial network includes the generative network and the discriminative network;

[0006] Receiving an image to be converted;

[0007] The image to be converted is processed based on the generative network to obtain a cartoonized target image; wherein the generative network is obtained based on unsupervised training combined with the discriminative network.

[0008] Optionally, the training of a generative adversarial network using real-life images and cartoon images includes:

[0009] Inputting the real scene image into the generation network to obtain a cartoonized first image;

[0010] Obtaining a first loss based on the real scene image and the first image;

[0011] Processing the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image;

[0012] Determining a second loss based on the second discrimination result and the third discrimination result;

[0013] Determine a generation loss based on the first loss and the second loss, and determine a determination loss based on the second loss;

[0014] The generative network is trained based on the generative loss, and the discriminative network is trained based on the discriminative loss.

[0015] Optionally, obtaining a first loss based on the real scene image and the first image includes:

[0016] Using a feature extraction network to extract features from the real scene image and the first image respectively, to obtain a real scene feature map and a first feature map;

[0017] The first loss is determined based on a difference between the real-scene feature map and the first feature map.

[0018] Optionally, before using a feature extraction network to extract features from the real scene image and the first image respectively, the method further includes:

[0019] Performing upsampling processing on the first image to obtain a second image having the same size as the real scene feature map;

[0020] The using a feature extraction network to extract features from the real scene image and the first image respectively includes:

[0021] The feature extraction network is used to extract features from the real scene image and the second image respectively.

[0022] Optionally, before processing the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image, the method further includes:

[0023] Performing superpixel segmentation processing on the first image and the cartoon image respectively to obtain a plurality of first sub-regions corresponding to the first image and a plurality of cartoon sub-regions corresponding to the cartoon image;

[0024] The step of processing the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image includes:

[0025] The plurality of first sub-regions and the plurality of cartoon sub-regions are processed based on the discriminant network to obtain a second discriminant result of whether the color distribution of each of the plurality of first sub-regions conforms to the cartoon color block distribution, and a third discriminant result of whether the color distribution of each of the plurality of cartoon sub-regions conforms to the cartoon color block distribution.

[0026] Optionally, determining a generation loss based on the first loss and the second loss comprises:

[0027] The generation loss is obtained based on a weighted sum of the first loss and the second loss.

[0028] Optionally, before determining a generation loss based on the first loss and the second loss and determining a discrimination loss based on the second loss, the method further includes:

[0029] Performing edge extraction processing on the first image and the cartoon image respectively to obtain first edge information corresponding to the first image and cartoon edge information corresponding to the cartoon image whose edge information is greater than a set threshold;

[0030] determining a third loss based on the first edge information and the cartoon edge information;

[0031] The step of determining a generation loss based on the first loss and the second loss, and determining a determination loss based on the second loss, comprises:

[0032] The generation loss is determined based on the first loss, the second loss, and the third loss, and the discrimination loss is determined based on the second loss and the third loss.

[0033] Optionally, the determining a third loss based on the first edge information and the cartoon edge information includes:

[0034] Processing the first edge information and the cartoon edge information based on the discriminant network to obtain a fourth discriminant result of whether the first edge information conforms to the cartoon edge distribution, and a fifth discriminant result of whether the cartoon edge information conforms to the cartoon edge distribution;

[0035] The third loss is determined based on the fourth discrimination result and the fifth discrimination result.

[0036] Optionally, determining the generation loss based on the first loss, the second loss, and the third loss, and determining the discrimination loss based on the second loss and the third loss, comprises:

[0037] Determine the generation loss based on a weighted sum of the first loss, the second loss, and the third loss;

[0038] The discrimination loss is determined based on a weighted sum of the second loss and the third loss.

[0039] According to another aspect of an embodiment of the present disclosure, a device for cartoonizing an image is provided, including:

[0040] A network training module, used to train a generative adversarial network using real-life images and cartoon images; wherein the generative adversarial network includes the generative network and the discriminative network;

[0041] An image receiving module, used for receiving an image to be converted;

[0042] A cartoonization module is used to process the image to be converted based on the generation network to obtain a cartoonized target image; wherein the generation network is obtained based on unsupervised training combined with the discriminant network.

[0043] Optionally, the network training module includes: a first loss unit, used to input the real scene image into the generation network to obtain a cartoonized first image; and obtain a first loss based on the real scene image and the first image;

[0044] A second loss unit is used to process the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image; and determine a second loss based on the second discriminant result and the third discriminant result;

[0045] A training unit is used to determine a generation loss based on the first loss and the second loss, and to determine a discrimination loss based on the second loss; and to train the generation network based on the generation loss, and to train the discrimination network based on the discrimination loss.

[0046] Optionally, the first loss unit is specifically used to use a feature extraction network to respectively extract features from the real scene image and the first image to obtain a real scene feature map and a first feature map; and determine the first loss based on the difference between the real scene feature map and the first feature map.

[0047] Optionally, the first loss unit is also used to perform upsampling processing on the first image to obtain a second image whose size is the same as that of the real scene feature map; and use the feature extraction network to extract features from the real scene image and the second image respectively.

[0048] Optionally, the second loss unit is further used to perform superpixel segmentation processing on the first image and the cartoon image, respectively, to obtain multiple first sub-regions corresponding to the first image and multiple cartoon sub-regions corresponding to the cartoon image; based on the discriminant network, the multiple first sub-regions and the multiple cartoon sub-regions are processed to obtain a second discrimination result of whether the color distribution of each of the multiple first sub-regions conforms to the cartoon color block distribution, and a third discrimination result of whether the color distribution of each of the multiple cartoon sub-regions conforms to the cartoon color block distribution.

[0049] Optionally, the training unit is specifically configured to obtain the generation loss based on a weighted sum of the first loss and the second loss.

[0050] Optionally, the network training module further includes:

[0051] A third loss unit is configured to perform edge extraction processing on the first image and the cartoon image respectively to obtain first edge information corresponding to the first image and cartoon edge information corresponding to the cartoon image whose edge information is greater than a set threshold; and determine a third loss based on the first edge information and the cartoon edge information:

[0052] The training unit is used to determine the generation loss based on the first loss, the second loss and the third loss, and determine the discrimination loss based on the second loss and the third loss.

[0053] Optionally, the third loss unit is specifically used to process the first edge information and the cartoon edge information based on the discriminant network to obtain a fourth discrimination result of whether the first edge information conforms to the cartoon edge distribution, and a fifth discrimination result of whether the cartoon edge information conforms to the cartoon edge distribution; based on the fourth discrimination result and the fifth discrimination result, determine the third loss.

[0054] Optionally, the training unit is specifically used to determine the generation loss based on a weighted sum of the first loss, the second loss and the third loss; and determine the discrimination loss based on a weighted sum of the second loss and the third loss.

[0055] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the image cartoonization method described in any of the above embodiments.

[0056] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0057] processor;

[0058] a memory for storing instructions executable by the processor;

[0059] The processor is used to read the executable instructions from the memory and execute the instructions to implement the image cartoonization method described in any one of the above embodiments.

[0060] According to another aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps of the image processing method described in any one of the embodiments are implemented.

[0061] Based on the above embodiments of the present disclosure, a method and device for image cartoonization, a storage medium, and an electronic device are provided to receive an image to be converted; the image to be converted is processed based on a generation network to obtain a cartoonized target image; wherein the generation network is unsupervisedly trained by combining with a discriminant network; and a deep learning method is used to realize cartoonization of real-life images in an end-to-end manner, with higher image conversion efficiency.

[0062] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0064] Figure 1 It is a flowchart of an image processing method provided by an exemplary embodiment of the present disclosure.

[0065] Figure 2 It is a flowchart of training a generative adversarial network in an image processing method provided by an exemplary embodiment of the present disclosure.

[0066] Figure 3 It is a flowchart of training a generative adversarial network in an image processing method provided by another exemplary embodiment of the present disclosure.

[0067] Figure 4 It is a schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present disclosure.

[0068] Figure 5 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0070] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0071] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0072] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0073] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0074] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0075] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0076] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0077] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0078] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0079] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0080] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0081] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0082] Application Overview

[0083] In the process of implementing the present disclosure, the inventors found that the existing image cartoonization methods have at least the following problems: the main content often has a certain degree of deformation and distortion.

[0084] Exemplary Methods

[0085] Figure 1 FIG. 1 is a flow chart of an image processing method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the following steps are included:

[0086] Step 102: training a generative adversarial network using real scene images and cartoon images.

[0087] Among them, the generative adversarial network includes a generative network and a discriminative network.

[0088] Optionally, the real scene images and cartoon images in this embodiment do not need to appear in pairs. It is only necessary to determine a large number of images with real scene content and cartoon content. The real scene corresponds to the cartoon. The inventor abstracted the two major characteristics of the cartoon image through analysis of the cartoon image: spatial color consistency (for example, the color value of all pixels in a color block is the same) and clear and clean edges. Since the real scene image is obtained by shooting a real scene, the color value of each pixel may be different, and the edges are not clear. In this embodiment, the real scene image is used as input by the generation network, and a cartoonized image is output. The discriminant network is then used to determine whether the image is a real cartoon image. In addition, the real cartoon image is added to the input end of the discriminant network, thereby improving the authenticity of the cartoonized image generated by the generation network.

[0089] Step 104: receiving an image to be converted.

[0090] The content of the image to be converted is a real-scene image, which may include but is not limited to people, objects, scenery, etc. The image to be converted may be obtained by shooting with a camera device.

[0091] Step 106: Process the image to be converted based on the generative network to obtain a cartoonized target image.

[0092] Among them, the generation network is obtained based on unsupervised training combined with the discriminant network.

[0093] In one embodiment, before use, the generative network in this embodiment is combined with the discriminative network to form a generative adversarial network, and the generative adversarial network is trained using unsupervised real images and cartoon images, so that the target image generated by the generative network is closer to the cartoon image.

[0094] The above-mentioned embodiment of the present disclosure provides an image processing method, which receives an image to be converted; processes the image to be converted based on a generative network to obtain a cartoonized target image; wherein the generative network is unsupervisedly trained by combining with a discriminative network; and utilizes a deep learning method to realize cartoonization of real-scene images in an end-to-end manner, with higher image conversion efficiency.

[0095] Optionally, the generation network (generator) provided in the above embodiment can adopt a common semantic segmentation framework of a general encode-decode structure. Optionally, this embodiment can adopt U-net, set the convolution layer with a step size of 2 for downsampling, and then use bilinear interpolation to achieve upsampling. The activation function adopts Leaky-ReLU, and the Adam optimization method is adopted. The initial learning rate can be set to 0.0001; the discriminant network (discriminator) adopts PatchGAN, and the last layer is the convolution layer. Each pixel output by the discriminator corresponds one-to-one to the pixel block in the corresponding receptive field of the input image. Its output reflects the discriminator's judgment on the category of the pixel block in the input image: that is, whether the pixel block belongs to a real image or a cartoon image. For example, the Adam optimization method is adopted, and the initial learning rate can be set to 0.0001.

[0096] like Figure 2 As shown, in the embodiment method provided by the present application, the generative adversarial network is trained using real scene images and cartoon images, including:

[0097] Step 202: input the real scene image into the generation network to obtain a cartoonized first image.

[0098] Step 204 , obtaining a first loss based on the real scene image and the first image.

[0099] In this embodiment, the first loss is the content loss of the first image obtained based on the generative network relative to the input real scene image, that is, the image content missing from the first image relative to the real scene image. The smaller the first loss is, the smaller the content loss of the first image generated by the generative network is. The training goal is to make the first loss as small as possible so that more content information in the real scene image can be retained in the first image obtained based on the generative network.

[0100] Step 206: Process the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image.

[0101] In this embodiment, the discriminant network is used to identify whether the input image is a real cartoon image, and the training goal is to enable the discriminant network to accurately identify whether the input image is an image generated by the generating network or a real cartoon image.

[0102] Step 208: Determine the second loss based on the second discrimination result and the third discrimination result.

[0103] Since the main difference between real-scene images and cartoon images lies in the distribution of color values ​​within color blocks, the second loss determined in this embodiment based on the second discrimination result and the third discrimination result is color block loss. The discriminator network trained by color block loss can distinguish the color block distribution of real-scene images and cartoon images.

[0104] Step 210 , determining a generation loss based on the first loss and the second loss, and determining a discrimination loss based on the second loss.

[0105] Optionally, the generated loss is obtained based on the weighted sum of the first loss and the second loss, and the specific weight value can be set according to the actual scenario, for example, 5:1, that is, when the weight value of the first loss is 5, the weight value of the second loss is 1.

[0106] Step 212: Train the generative network based on the generative loss, and train the discriminative network based on the discriminative loss.

[0107] This embodiment uses the generation loss to train the generation network, and trains the discriminative network based on the discriminative loss, that is, the discriminative loss is back-propagated through the discriminative network, and the first loss is added before back-propagating to the generation network. The network parameters in the generation network are adjusted in combination with the first loss and the second loss, so that the trained generation network is closer to the actual cartoon image in terms of content and color block distribution.

[0108] Optionally, step 204 in the above embodiment may include:

[0109] Using a feature extraction network to extract features from the real scene image and the first image respectively, to obtain a real scene feature map and a first feature map;

[0110] Based on the difference between the actual scene feature map and the first feature map, a first loss is determined.

[0111] Optionally, a feature extraction network (e.g., VGG-net) is used to extract high-order content features of the input real scene image and the first image, and the average absolute distance between the two is minimized to ensure that the main content is not distorted. Specifically, the first loss can be determined by the following formula (1);

[0112] L tv =||VGG(G(I p ))-VGG(I p )|| Formula (1)

[0113] Among them, I p Represents a real scene image; G(I p ) represents the first image obtained by the generative network based on the real scene image; VGG(I p) represents the real scene feature map obtained by extracting features from the real scene image using the feature extraction network; VGG(G(I p )) represents a first feature map obtained by extracting features from the first image using a feature extraction network; L tv In addition, in order to determine the difference between the real scene feature map and the first feature map, it is necessary to process the size of the first image to be the same as the real scene image before calculating the difference, so as to ensure that the first feature map and the real scene feature map have the same size; optionally, upsampling is performed on the first image to obtain a second image with the same size as the real scene feature map; at this time,

[0114] The feature extraction network is used to extract features from the real scene image and the first image respectively, including:

[0115] The feature extraction network is used to extract features from the real scene image and the second image respectively.

[0116] By performing feature extraction on the real scene image and the second image of the same size as the real scene image through the feature extraction network, a real scene feature map and a first feature map of the same size can be obtained, thereby ensuring that the first loss can be determined based on each feature point in the feature map.

[0117] In the above Figure 2 In the illustrated embodiment, before executing step 206, the following steps may also be included:

[0118] Superpixel segmentation processing is performed on the first image and the cartoon image respectively to obtain a plurality of first sub-regions corresponding to the first image and a plurality of cartoon sub-regions corresponding to the cartoon image.

[0119] Image segmentation (also known as superpixel segmentation) refers to the process of subdividing a digital image into multiple image sub-regions (a collection of pixels). A superpixel is a small area consisting of a series of pixels that are adjacent in position and have similar characteristics such as color, brightness, and texture.

[0120] At this time, step 206 may include:

[0121] Based on the discriminant network, the multiple first sub-regions and the multiple cartoon sub-regions are processed to obtain a second discriminant result of whether the color distribution in each of the multiple first sub-regions conforms to the cartoon color block distribution, and a third discriminant result of whether the color distribution in each of the multiple cartoon sub-regions conforms to the cartoon color block distribution.

[0122] In this embodiment, a superpixel segmentation method is used to extract the color block distribution of the first image and the cartoon image, and then the processed images are simultaneously sent to the discriminant network to train the discriminant network so that it can distinguish the color block distribution of the real image and the cartoon image. The second loss determined by the color block distribution can be determined based on the following formula (2):

[0123] L cl =logD(F sp (I c , I c ))+log(1-D(F sp (G(I p ), G(I p )))) Formula (2)

[0124] Among them, I p I represents the real scene image; c Represents a cartoon image; G(I p ) represents the first image obtained by the generative network based on the real scene image; F sp (I c , I c ) represents the result of superpixel segmentation of cartoon image; F sp (G(I p ), G(I p ) represents the result of superpixel segmentation of the first image; D() represents the processing of the data in the brackets using the discriminant network. In this embodiment, D(F sp (I c , I c )) represents the third discrimination result, D(F sp (G(I p ), G(I p ))) represents the second discrimination result; L cl Indicates the second loss.

[0125] like Figure 3 As shown, in the embodiment method provided by the present application, the generative adversarial network is trained using real scene images and cartoon images, including:

[0126] Step 302: input the real scene image into the generation network to obtain a cartoonized first image.

[0127] Step 304: Obtain a first loss based on the real scene image and the first image.

[0128] In this embodiment, the first loss is the content loss of the first image obtained based on the generative network relative to the input real scene image, that is, the image content missing from the first image relative to the real scene image. The smaller the first loss is, the smaller the content loss of the first image generated by the generative network is. The training goal is to make the first loss as small as possible so that more content information in the real scene image can be retained in the first image obtained based on the generative network.

[0129] Step 306: Process the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image.

[0130] In this embodiment, the discriminant network is used to identify whether the input image is a real cartoon image, and the training goal is to enable the discriminant network to accurately identify whether the input image is an image generated by the generating network or a real cartoon image.

[0131] Step 308: Determine the second loss based on the second discrimination result and the third discrimination result.

[0132] Since the main difference between real-scene images and cartoon images lies in the distribution of color values ​​within color blocks, the second loss determined in this embodiment based on the second discrimination result and the third discrimination result is color block loss. The discriminator network trained by color block loss can distinguish the color block distribution of real-scene images and cartoon images.

[0133] Step 310, performing edge extraction processing on the first image and the cartoon image respectively to obtain first edge information corresponding to the first image and cartoon edge information corresponding to the cartoon image whose edge information is greater than a set threshold; and determining a third loss based on the first edge information and the cartoon edge information.

[0134] Step 312, determining a generation loss based on the first loss, the second loss and the third loss, and determining a discrimination loss based on the second loss and the third loss.

[0135] Optionally, the generation loss is obtained based on the weighted sum of the first loss, the second loss and the third loss, and the specific weight values ​​can be set according to the actual scenario, for example, 1000:200:1, that is, when the weight value of the first loss is 1000, the weight value of the second loss is 200, and the weight value of the third loss is 1; and for the discriminant network, the discriminant loss can be obtained based on the weighted sum of the second loss and the third loss, and the weight value can be set according to the actual scenario, for example, 1:1, that is, the sum of the second loss and the third loss is the discriminant loss.

[0136] Step 314, training the generative network based on the generative loss, and training the discriminative network based on the discriminative loss.

[0137] In addition to the difference in color block distribution, the real scene image and the cartoon image also have a difference in whether the edges are clear. In this embodiment, a third loss is added to make the edges of the cartoonized target image generated by the trained generation network clear and clean, which is closer to the real cartoon image. The setting of the weights between the three losses can further improve the performance of the generation network provided in the embodiment of the present application. If the weight of the content loss is too large, the cartoon style of the image will be weakened. Otherwise, the main content will be distorted while the cartoon style is increased. In this embodiment, appropriate weights are set for the three losses to ensure that the content is not distorted while ensuring cartoonization.

[0138] In the above Figure 3 Based on the illustrated embodiment, step 310 may include:

[0139] Processing the first edge information and the cartoon edge information based on the discriminant network to obtain a fourth discriminant result of whether the first edge information conforms to the cartoon edge distribution, and a fifth discriminant result of whether the cartoon edge information conforms to the cartoon edge distribution;

[0140] Based on the fourth discrimination result and the fifth discrimination result, the third loss is determined.

[0141] In this embodiment, optionally, the edges of the first image and the cartoon image can be extracted by a Sobel operator. Considering the consistency of the cartoon image content, only relatively obvious edge information is considered in this embodiment. Optionally, a threshold value of 0.5 can be set. The edges of the first image and the cartoon image are extracted by the set threshold value. The pixel values ​​greater than the set threshold value in the values ​​determined by the Sobel operator are input into the discriminant network as edge information. The discriminant network is trained to enable it to distinguish the edge information of the first image and the cartoon image. Optionally, the third loss can be determined based on the following formula (3):

[0142] L eg =logD(F edge (I c , I c )>0.5)+log(1-D(F edge (G(I p ), G(I p ))>0.5))Formula (3)

[0143] Among them, I p I represents the real scene image; c Represents a cartoon image; G(I p ) represents the first image obtained by the generative network based on the real scene image; F edge (I c , I c)>0.5 represents edge information obtained by edge extraction of the cartoon image (the threshold value in this embodiment is 0.5); (G(I p ), G(I p ))>0.5 represents the edge information obtained by edge extraction of the first image (the threshold value in this embodiment is 0.5); D() represents the processing of the data in the brackets by using the discriminant network. In this embodiment, D(F edge (G(I p ), G(I p ))>0.5) represents the fourth discrimination result, D(F edge (I c , I c )>0.5) indicates the fifth discrimination result; L eg Indicates the third loss.

[0144] Any image cartoonization method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any image cartoonization method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any image cartoonization method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0145] Exemplary Devices

[0146] Figure 4 FIG. 1 is a schematic diagram of a structure of an image cartoonization device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the device provided in this embodiment includes:

[0147] The network training module 41 is used to train the generative adversarial network using real scene images and cartoon images.

[0148] Among them, the generative adversarial network includes a generative network and a discriminative network.

[0149] The image receiving module 42 is used to receive the image to be converted.

[0150] The cartoonization module 43 is used to process the image to be converted based on the generation network to obtain a cartoonized target image.

[0151] Among them, the generation network is obtained based on unsupervised training combined with the discriminant network.

[0152] The above-mentioned embodiment of the present disclosure provides an image processing device, which receives an image to be converted; processes the image to be converted based on a generation network to obtain a cartoonized target image; wherein the generation network is unsupervisedly trained by combining with a discriminant network; and utilizes a deep learning method to realize cartoonization of real-scene images in an end-to-end manner, with higher image conversion efficiency.

[0153] In some optional embodiments, the network training module includes:

[0154] The first loss unit is used to input the real scene image into the generation network to obtain a cartoonized first image; and obtain a first loss based on the real scene image and the first image.

[0155] The second loss unit is used to process the first image and the cartoon image based on the discriminant network to obtain a second discriminant result corresponding to the first image and a third discriminant result corresponding to the cartoon image; based on the second discriminant result and the third discriminant result, determine the second loss.

[0156] The training unit is used to determine the generation loss based on the first loss and the second loss, and determine the discrimination loss based on the second loss; the generation network is trained based on the generation loss, and the discrimination network is trained based on the discrimination loss.

[0157] Optionally, the first loss unit is specifically used to use a feature extraction network to extract features from the real scene image and the first image respectively to obtain a real scene feature map and a first feature map; and determine the first loss based on the difference between the real scene feature map and the first feature map.

[0158] Optionally, the first loss unit is further used to perform upsampling processing on the first image to obtain a second image having the same size as the real scene feature map; and use a feature extraction network to extract features from the real scene image and the second image respectively.

[0159] Optionally, the second loss unit is further used to perform superpixel segmentation processing on the first image and the cartoon image, respectively, to obtain multiple first sub-regions corresponding to the first image and multiple cartoon sub-regions corresponding to the cartoon image; based on the discriminant network, the multiple first sub-regions and the multiple cartoon sub-regions are processed to obtain a second discrimination result of whether the color distribution of each of the multiple first sub-regions conforms to the cartoon color block distribution, and a third discrimination result of whether the color distribution of each of the multiple cartoon sub-regions conforms to the cartoon color block distribution.

[0160] Optionally, the training unit is specifically configured to obtain a generation loss based on a weighted sum of the first loss and the second loss.

[0161] Optionally, the network training module further includes:

[0162] A third loss unit is used to perform edge extraction processing on the first image and the cartoon image respectively, to obtain first edge information corresponding to the first image whose edge information is greater than a set threshold and cartoon edge information corresponding to the cartoon image; and to determine a third loss based on the first edge information and the cartoon edge information;

[0163] A training unit is used to determine a generation loss based on the first loss, the second loss and the third loss, and to determine a discrimination loss based on the second loss and the third loss.

[0164] Optionally, the third loss unit is specifically used to process the first edge information and the cartoon edge information based on the discriminant network to obtain a fourth discrimination result of whether the first edge information conforms to the cartoon edge distribution, and a fifth discrimination result of whether the cartoon edge information conforms to the cartoon edge distribution; based on the fourth discrimination result and the fifth discrimination result, determine the third loss.

[0165] Optionally, the training unit is specifically used to determine the generation loss based on the weighted sum of the first loss, the second loss and the third loss; and determine the discrimination loss based on the weighted sum of the second loss and the third loss.

[0166] Exemplary Electronic Devices

[0167] Below, reference Figure 5 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device 100 and the second device 200, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0168] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0169] like Figure 5 As shown, the electronic device 50 includes one or more processors 51 and a memory 52 .

[0170] The processor 51 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 50 to perform desired functions.

[0171] The memory 52 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 51 may run the program instructions to implement the image cartoonization method of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0172] In one example, the electronic device 50 may further include: an input device 53 and an output device 54, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0173] For example, when the electronic device is the first device 100 or the second device 200, the input device 53 may be the microphone or microphone array described above, for capturing input signals from a sound source. When the electronic device is a stand-alone device, the input device 53 may be a communication network connector, for receiving collected input signals from the first device 100 and the second device 200.

[0174] In addition, the input device 53 may also include, for example, a keyboard, a mouse, etc.

[0175] The output device 54 can output various information to the outside, including the determined distance information, direction information, etc. The output device 54 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0176] Of course, to simplify, Figure 5 Only some of the components related to the present disclosure in the electronic device 50 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 50 may also include any other appropriate components.

[0177] Exemplary computer program products and computer-readable storage media

[0178] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method of image cartoonization according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0179] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0180] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for image cartoonization according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0181] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0182] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0183] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0184] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0185] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0186] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0187] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0188] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An image processing method, characterized in that: include: Using real-life images and cartoon images to train a generative adversarial network; wherein the generative adversarial network includes a generative network and a discriminative network; the real-life images and the cartoon images are not paired; Receiving an image to be converted; The image to be converted is processed based on the generative network to obtain a cartoonized target image; wherein the generative network is obtained based on unsupervised training combined with the discriminative network; The method of training a generative adversarial network using real-life images and cartoon images includes: Inputting the real scene image into the generation network to obtain a cartoonized first image; A first loss is obtained based on the real scene image and the first image; the first loss is a content loss; Performing superpixel segmentation processing on the first image and the cartoon image respectively to obtain a plurality of first sub-regions corresponding to the first image and a plurality of cartoon sub-regions corresponding to the cartoon image; Processing the plurality of first sub-regions and the plurality of cartoon sub-regions based on the discriminant network to obtain a second discriminant result of whether the color distribution in each of the plurality of first sub-regions conforms to the cartoon color block distribution, and a third discriminant result of whether the color distribution in each of the plurality of cartoon sub-regions conforms to the cartoon color block distribution; Based on the second discrimination result and the third discrimination result, determining a second loss; the second loss is a color block loss; Determine a generation loss based on the first loss and the second loss, and determine a determination loss based on the second loss; The generative network is trained based on the generative loss, and the discriminative network is trained based on the discriminative loss.

2. The method according to claim 1, characterized in that The obtaining a first loss based on the real scene image and the first image includes: Using a feature extraction network to extract features from the real scene image and the first image respectively, to obtain a real scene feature map and a first feature map; The first loss is determined based on a difference between the real-scene feature map and the first feature map.

3. The method according to claim 2, characterized in that Before using the feature extraction network to extract features from the real scene image and the first image respectively, the method further includes: Performing upsampling processing on the first image to obtain a second image having the same size as the real scene feature map; The using a feature extraction network to extract features from the real scene image and the first image respectively includes: The feature extraction network is used to extract features from the real scene image and the second image respectively.

4. The method according to any one of claims 1 to 3, characterized in that: The determining of generating loss based on the first loss and the second loss comprises: The generation loss is obtained based on a weighted sum of the first loss and the second loss.

5. The method according to any one of claims 1 to 3, characterized in that: Before determining the generation loss based on the first loss and the second loss and determining the discrimination loss based on the second loss, the method further includes: Performing edge extraction processing on the first image and the cartoon image respectively to obtain first edge information corresponding to the first image and cartoon edge information corresponding to the cartoon image whose edge information is greater than a set threshold; determining a third loss based on the first edge information and the cartoon edge information; The step of determining a generation loss based on the first loss and the second loss, and determining a determination loss based on the second loss, comprises: The generation loss is determined based on the first loss, the second loss, and the third loss, and the discrimination loss is determined based on the second loss and the third loss.

6. The method according to claim 5, characterized in that The determining a third loss based on the first edge information and the cartoon edge information includes: Processing the first edge information and the cartoon edge information based on the discriminant network to obtain a fourth discriminant result of whether the first edge information conforms to the cartoon edge distribution, and a fifth discriminant result of whether the cartoon edge information conforms to the cartoon edge distribution; determining the third loss based on the fourth discrimination result and the fifth discrimination result; and / or The step of determining the generation loss based on the first loss, the second loss, and the third loss, and determining the identification loss based on the second loss and the third loss, comprises: Determine the generation loss based on a weighted sum of the first loss, the second loss, and the third loss; The discrimination loss is determined based on a weighted sum of the second loss and the third loss.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the image processing method described in any one of claims 1 to 6.

8. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

Images