Cartoon model construction method, apparatus, electronic device, storage medium and program

Abstract

The present application discloses a cartoonization model construction method, apparatus, electronic device, storage medium and program, the method including: employing a pre-trained first generative model to generate a predetermined number of sample true drawings; constructing a second generative model based on the first generative model, and employing the second generative model to generate sample cartoon drawings corresponding to each sample true drawing; combining each sample true drawing and the corresponding sample cartoon drawing into a sample image pair; and based on a sample set composed of a plurality of the sample image pairs, using weights corresponding to the second generative model as initial weights, fitting to a pre-set initial cartoonization model, and generating a cartoonization model for converting a target image into a full-picture cartoonization image.

Description

[Technical field]

[0001] The present application, This is the Japanese national phase of an international application with application number PCT / CN2022 / 132131 filed on November 16, 2022, November 16, 2021 On the day Priority is claimed from the filed Chinese patent application No. 202111356773.X, the entire contents of which are incorporated herein by reference.

[0002] This application relates to the technical field of image processing. Regarding , for example, a method for constructing a cartoon model, an apparatus for constructing a cartoon model, Build a cartoon model The present invention relates to an electronic device, a computer-readable storage medium and a computer program product. [Background technology]

[0003] Image cartooning technology is one of the common tasks of image editing in computer vision. It is very widely applied in life and entertainment, such as filmmaking, animation production, short videos, live broadcasts and other scenes, where cartooning processing is performed on images. Summary of the Invention

[0004] To the original wish , comics Imaging model construction method and device Electronic building cartoon model A device, a storage medium and a program product are provided. [Means for solving the problem]

[0005] Books In one embodiment of the present application, there is provided a method for constructing a cartoonized model, the method comprising: Employing a pre-trained first generative model to generate a preset number of sample true drawings; constructing a second generative model based on the first generative model, and employing the second generative model to generate sample cartoon drawings corresponding to each sample true drawing; combining each of the sample true images and a corresponding sample cartoon image into a sample image pair; Based on a sample set consisting of a plurality of the sample image pairs, weights corresponding to the second generative model are set as initial weights, and fitting is performed to an initial cartoon model set in advance to generate a cartoon model for converting the target image into a full-picture cartoon image; Includes.

[0006] Books In accordance with an embodiment of the present invention, there is also provided an apparatus for constructing a cartoonized model, the apparatus comprising: The method includes a sample true image generation module, a sample cartoon image generation module, an image pairing module, and a cartoon model fitting module; The sample true drawing generation module is configured to generate a preset number of sample true drawings by employing a first generative model that is pre-trained; The sample cartoon diagram generation module is configured to construct a second generative model based on the first generative model, and employ the second generative model to generate a sample cartoon diagram corresponding to each sample true diagram; the image pairing module is configured to pair each of the sample true images with a corresponding sample cartoon image into a sample image pair; The cartoon model fitting module is configured to use weights corresponding to the second generative model as initial weights based on a sample set consisting of a plurality of the sample image pairs, and perform fitting to a pre-installed initial cartoon model to generate a cartoon model for converting a target image into a full-picture cartoon image.

[0007] Books In the embodiment of the present application Build a cartoon model An electronic device is also provided, the electronic device comprising: one or more processors and a memory device; the storage device is configured to store one or more programs; When the one or more programs are executed by the one or more processors, The Realize the method.

[0008] Books In an embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program, and when the program is executed by a processor, The Realize the method.

[0009] Books In accordance with an embodiment of the present invention, there is also provided a computer program product, the computer program product including computer executable instructions which, when executed, implement the method of the first or second aspect. [Brief description of the drawings]

[0010] [Figure 1] 1 is a flowchart of an embodiment of a method for constructing a cartoonized model provided in one embodiment of the present application. [Diagram 2] FIG. 2 is a schematic diagram illustrating the effect of performing full-image cartooning processing on an image based on the cartooning model provided in one embodiment of the present application; [Diagram 3] FIG. 1 is a schematic diagram of a model architecture of a StyleGAN2 model provided in one embodiment of the present application. [Figure 4] 4 is a flowchart of an embodiment of a method for constructing a cartoonized model provided in another embodiment of the present application. [Diagram 5] 4 is a flowchart of an embodiment of a method for constructing a cartoonized model provided in another embodiment of the present application. [Figure 6] 4 is a flowchart of an embodiment of a method for constructing a cartoonized model provided in another embodiment of the present application. [Figure 7] FIG. 1 is a schematic diagram of an initial cartooning model architecture provided in one embodiment of the present application. [Figure 8] 4 is a flowchart of an embodiment of a method for constructing a cartoonized model provided in another embodiment of the present application. [Figure 9] FIG. 2 is a structural block diagram of an embodiment of an apparatus for constructing a cartoonized model provided in one embodiment of the present application. [Figure 10]FIG. 1 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the related art, the implementation method of the image cartoonization process is as follows.

[0012] One is a processing method based on basic signals, which mainly builds a material library, and matches the most suitable material in the material library according to various related basic signals such as height, body weight, hair color, clothing color, etc., and then combines the matched material into an animated image. This method has the disadvantages of a single image, lack of flexibility, lack of diversity, and poor user feature similarity.

[0013] The other is a processing method for making a face by pasting an image. This method transforms a real human face into a cartoon face shape by deformation, and then pastes various materials such as cartoon faces, eyes, eyebrows, etc. to realize the image cartoonization. However, the effect of this method is single, and the animated images created by different people are largely similar, poor in diversity, weak in cartoon feeling, and poor in realism.

[0014] 1 is a flowchart of an embodiment of a method for constructing a cartoon model provided in an embodiment of the present application, which can be realized by a cartoon model construction device, where the cartoon model construction device can be installed in a server or a client, and the embodiment is not limited thereto. The method aims to avoid the situation in related technologies where the cartoon image generated during cartoonization processing is uniform, lacks flexibility, lacks diversity, has poor similarity in user characteristics, and has a weak cartoon feel.

[0015] The cartoon model constructed in this embodiment is applicable to scenes such as image processing, short video, film production, live broadcast, 3D cartoon, etc., and can be used to process images in scenes such as image processing, short video, film, live broadcast, etc. into full-picture cartoon images. For example, as shown in Fig. 2, an image is given, the image is input to the cartoon model, and the cartoon model can output an image after the full picture is cartoonized (i.e., the full-picture cartoon image), and the content of the full-picture cartoon image and the input image is the same, but it has the style of a cartoon, that is, all the elements in the input image are cartoonized. In addition, a short video is given, and the images of each frame of the short video are input to the cartoon model, and the cartoon model can output a full-picture cartoon image corresponding to the images of multiple frames, thereby realizing the full-picture cartoonization of the images of each frame in the input short video.

[0016] The cartoon model constructed in this embodiment is accessed through an APP or a Web page according to the development documents.

[0017] As shown in FIG. 1, the embodiment may include the following steps:

[0018] In step 110, a pre-trained first generative model is employed to generate a preset number of sample diagrams.

[0019] A generative model is an important model in probability statistics and machine learning, and refers to a model used to randomly generate observable data. For example, the first generative model may be a StyleGAN2 model, and the StyleGAN2 model may be adopted to randomly generate a preset number of sample true images. Here, the sample true images may be images that are not cartoonized, for example, including images of real people.

[0020] GAN (Generative Adversarial Networks) is a type of deep learning model that can generate new content. StyleGAN is one of GANs, a generative model that uses style. StyleGAN is an advanced high-resolution image synthesis method that has been proven to work reliably on a variety of datasets. StyleGAN can be used to generate not only realistic human figures, but also other animals, cars, and even rooms. However, StyleGAN is not perfect, and its most obvious flaw is that the generated images may contain artifacts such as spots, but this flaw is overcome by StyleGAN2, improving the quality of the generated images.

[0021] As shown in the model architecture of the StyleGAN2 model in Figure 3, StyleGAN2 consists of two parts, including the left part which is a mapping network in Figure 3 and the right part which is a synthesis network.

[0022] The Mapping Network can unwrap the input more appropriately. As shown in Figure 3, the Mapping Network is composed of eight fully connected layers (FC), whose input is Gaussian noise (latent Z), and the hidden variables (W) are obtained through the Mapping Network.

[0023] The synthesis network is composed of modules such as a learnable affine transformation A, a modulation module Mod-Demod, and an upsampling Upsample. In addition, the synthesis network further includes weights (w), deviations (b), and constant inputs (c, i.e., Const 4*4*512, which indicates a learnable constant), and the activation function (Leaky ReLU) is always applied immediately after adding the bias.

[0024] Here, the learnable affine transformation A is constructed by a single fully connected layer, and Upsample can perform the upsampling operation using deconvolution (also called transposed convolution).

[0025] The processing flow of the modulation module Mod-Demod is as follows:

number

number

number

[0026] In FIG. 3, the rightmost one is the injection of random noise, and B is a learnable noise parameter. The injection of random noise is used to make the generated image more true and realistic. For example, when generating these noises, subtle features of a human face, such as facial blemishes, can be generated.

[0027] In step 120, a second generative model is constructed based on the first generative model, and the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0028] In this step, the second generative model may also be a StyleGAN2 model, and the model architecture of the first and second generative models is the same, but the difference is that the model weights are different due to different training goals. The training model of the first generative model is for generating sample true figures, i.e., images that are not cartoonized. The training model of the second generative model is for generating sample cartoon figures, i.e., images after cartoonization.

[0029] In one embodiment, the weights of the pre-trained first generative model can be fine-tuned using cartoon data as training target to obtain a second generative model for cartoonization, and then the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0030] In step 130, the sample true image and the corresponding sample cartoon image are combined into a sample image pair.

[0031] In this step, after generating a preset number of sample true images and corresponding sample cartoon images, each sample true image and its corresponding sample cartoon image can be organized into a sample image pair (picture to picture, P2P), and all the sample image pairs are assembled into a sample set for subsequent model fitting.

[0032] In addition, the preset number may be a kind of empirical value, and the specific value of the preset number can be determined based on the required accuracy of the model. For example, the preset number may be 150,000, that is, 150,000 sample image pairs are generated.

[0033] In step 140, based on a sample set consisting of a plurality of the sample image pairs, weights corresponding to the second generative model are used as initial weights, and fitting is performed against a pre-established initial cartoonization model to generate a cartoonization model for converting the target image into a full-picture cartoonization image.

[0034] In this step, after obtaining a sample set consisting of sample image pairs, the sample true images in the sample set are used as training data, the weights corresponding to the second generation model to be finally generated are used as initial weights, and the sample cartoon images corresponding to each sample true image are used as optimization targets. Fitting is performed to the initial cartoon model set up in advance, and a finally fitted cartoon model is obtained, which can be set up to transform the target image into a full-picture cartoon image.

[0035] In this embodiment, when constructing a cartoonization model for converting a target image into a full-picture cartoonized image, a first generation model is first trained to randomly generate a preset number of sample true images, and then a second generation model is constructed based on the first generation model to generate cartoon images, and the second generation model is used to generate sample cartoon images corresponding to each sample true image, and the sample true images and the corresponding sample cartoon images are combined into sample image pairs to obtain a sample set, and then the weights corresponding to the second generation model are used as initial weights, and the sample set is used to perform fitting to the pre-set initial cartoonization model, and the fitted model is a cartoonization model, which can realize the full-picture cartoonization process. The second generation model in this embodiment is related to the first generation model, and the weights of the second generation model are used as initial weights of the cartoonization model, and the image pairs are used to obtain the image pairs as training data to realize fitting to the cartoonization model, and the finally obtained cartoonization model has higher robustness and generalizability, which improves the effect of full-picture cartoonization. In addition, this embodiment requires less data than other methods.

[0036] 4 is a flowchart of a method for constructing a cartoon model according to another embodiment of the present application, which is based on the above embodiment and describes the construction process of the second generative model in more detail. As shown in FIG. 4, the present embodiment can include the following steps:

[0037] In step 410, a pre-trained first generative model is employed to generate a preset number of sample diagrams.

[0038] In step 420, adjustments are made to the weights of the first generative model to generate an intermediate cartoon model.

[0039] In this embodiment, the training target of the first generation model is the original drawing that is not cartoonized, and the training target of the intermediate cartoon model is the cartoon image after cartoonization processing, so the weight of the first generation model is the initial weight of the intermediate cartoon model, and the cartoon image is used as the training target to generate the intermediate cartoon model. In this way, the weight of the intermediate cartoon model is obtained after adjusting the weight of the first generation model.

[0040] In step 430, weights corresponding to a portion of a specified layer in the intermediate cartoon model are replaced with weights corresponding to the portion of the specified layer in the first generative model, and weight interpolation is performed to generate a second generative model.

[0041] In order to ensure that some attributes of the final output cartoon image match those in the original drawing generated by the first generative model, after generating the intermediate cartoon model, weights corresponding to some specified layers in the intermediate cartoon model can be replaced with weights corresponding to those specified layers in the first generative model, and weight interpolation can be performed to generate a second generative model.

[0042] For example, some of the designated layers include at least one of a layer for controlling a person's posture and a layer for controlling a person's skin color, that is, in order to ensure that the posture and skin color of a person after cartoonization are consistent with those of a real person in the original drawing, after obtaining the intermediate cartoon model, the weights of the layer for controlling a person's posture and the layer for controlling a person's skin color in the intermediate cartoon model are replaced with the weights of the layer for controlling a person's posture and the layer for controlling a person's skin color in the first generative model, and weight interpolation is performed in the intermediate cartoon model, so as to finally obtain the weights of the second generative model.

[0043] Weighted interpolation refers to using an interpolation algorithm to calculate a new weight between two weights, and inserting the new weight between the two weights. This embodiment does not limit the specific interpolation algorithm of weighted interpolation, and includes, for example, an inverse distance weighted interpolation (IDW) to perform weighted interpolation. Inverse distance weighted interpolation can also be called distance inverse power method, and distance inverse power grid mesh method is a weighted average interpolation method, which means that it can perform proper or smooth manner interpolation. The explicit assumption of inverse distance weighted (IDW) interpolation is that things that are relatively close to each other are more similar than things that are relatively far from each other. When predicting values ​​for a position where nothing is measured, the inverse distance weighting method can adopt measurements around the predicted position. Measurements that are closest to the predicted position have a larger influence on the predicted value than measurements that are relatively far from the predicted position. The inverse distance weighting method assumes that each measurement point has a partial influence, and this influence decreases as the distance increases. This method is called inverse distance weighting because the weights assigned to points with closest distance predicted positions are relatively large, but the weights decrease as a function of distance.

[0044] In addition, those skilled in the art can also adopt weight interpolation based on covariance weight interpolation algorithm, Krikin interpolation method, etc.

[0045] In step 440, the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0046] In step 450, the sample true image and the corresponding sample cartoon image are combined into a sample image pair.

[0047] In step 460, based on a sample set consisting of a plurality of the sample image pairs, weights corresponding to the second generative model are used as initial weights, and fitting is performed against a pre-established initial cartoonization model to generate a cartoonization model for converting the target image into a full-picture cartoonization image.

[0048] In this embodiment, when constructing the second generative model, the first generative model trained in advance is used as a basis, the cartoon image is used as a training target, the weights of the first generative model are adjusted to obtain an intermediate cartoon model, and then the weights of some layers of the intermediate cartoon model are replaced and the weights are interpolated to obtain the final weights of the second generative model, thus completing the construction of the second generative model. Compared with simply training the second generative model with the cartoonized image as a training target, the second generative model constructed by the above method of this embodiment is more robust and improves the authenticity of the cartoonized image.

[0049] 5 is a flowchart of a method for constructing a cartoon model according to another embodiment of the present application, which is based on the above embodiment and describes the training sample processing process in more detail. As shown in FIG. 5, the present embodiment includes the following steps:

[0050] In step 510, a pre-trained first generative model is employed to generate a preset number of sample true drawings.

[0051] In step 520, a second generative model is constructed based on the first generative model, and the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0052] In step 530, the sample true image and the corresponding sample cartoon image are combined into a sample image pair.

[0053] In step 540, data augmentation is performed on a sample set based on a plurality of the sample image pairs, where the data augmentation is performed by randomly augmenting the sample true and cartoon drawings. M This includes performing at least one of the following: rotate, random crop, random enlarge, random shrink.

[0054] In this step, all sample image pairs are assembled into a sample set, and then data augmentation is performed on the sample true images and sample cartoon images in the sample set to increase the amount of training data and improve the robustness and generalization ability of the model.

[0055] In implementation, data augmentation may include at least one of the following: various noise augmentations, augmentations such as downsampling then upsampling, data augmentation for sample image pairs, and the like.

[0056] Exemplarily, data augmentation for the sample image pairs may include randomly augmenting the sample true and / or cartoon images. M This includes but is not limited to rotating, randomly cropping, randomly scaling up, randomly shrinking, etc.

[0057] In step 550, based on the sample set, weights corresponding to the second generative model are used as initial weights, and fitting is performed against a pre-established initial cartoonization model to generate a cartoonization model for converting the target image into a full-view cartoonization image.

[0058] After obtaining a sample set through the data augmentation method, the sample set can be used to set the weights corresponding to the second generation model as initial weights, and fitting can be performed to the initial cartoon model set in advance to generate a cartoon model.

[0059] In this embodiment, after obtaining sample image pairs combining a sample true image and a corresponding sample cartoon image, the sample image pairs are assembled into a sample set as training data. Then, various methods of data expansion are adopted for the sample set, and a cartoonization model is adopted to train the sample set after data expansion to realize full-picture cartoonization technology. The robustness of the cartoonization model is further improved, the model is robust to objects at any angle (e.g., human objects), and has strong generalizability to various scenes. The full-picture cartoonization effect for various low-quality images is still relatively good.

[0060] 6 is a flowchart of a method for constructing a cartoon model provided in another embodiment of the present application, which describes the cartoon model construction process in more detail based on the above embodiment. As shown in FIG. 6, the present embodiment can include the following steps:

[0061] In step 610, a pre-trained first generative model is employed to generate a preset number of sample diagrams.

[0062] In step 620, a second generative model is constructed based on the first generative model, and the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0063] In step 630, the sample true image and the corresponding sample cartoon image are combined into a sample image pair.

[0064] In step 640, an encoder in a pre-installed initial cartoon model is employed to perform feature extraction on the sample images in the sample set to obtain corresponding feature images and style attribute information, and the feature images and the style attribute information are output to a decoder of the initial cartoon model.

[0065] In one embodiment, the initial cartoon model may include an Encoder and a Decoder, and as shown in Figure 7, the left dashed box part is the Encoder, and the right dashed box part is the Decoder. The function of the Encoder is to perform information extraction on a plurality of sample real images, and output the extracted feature images and style attribute information to the Decoder, which combines the feature images and style attribute information to output a full-image cartoon image.

[0066] The initial weights of the encoder in this embodiment are the weights of the encoder that has previously edited various real human images.

[0067] In one embodiment, the structure of the Encoder may include an input layer, several residual layers, and a fully connected layer, in which the multiple residual layers are used for extracting feature maps in the sample truth maps and outputting the feature maps to corresponding layers of the decoder, and the fully connected layer is used for extracting style attribute information of the sample truth maps and outputting the style attribute information to multiple layers of the decoder.

[0068] For example, the structure of the Encoder is shown in Table 1 below. In Table 1, there are five residual layers (ResBlock), and the size of the feature map output by each residual layer is specified, such as 512*512*3, 256*256*32, etc. in Table 1. The fully connected layer FC outputs style attribute information of size 16*512.

[0069] [Table 1]

[0070] As shown in Figure 7, the feature map extracted by each residual layer needs to be output to the next layer for processing, and then output to the corresponding layer of the Decoder (except for the last residual layer, which outputs only the result to the corresponding layer of the Decoder). The corresponding layer here is a decoding layer that matches the size of the currently output feature map. For example, if the size of the currently output feature map is 32*32*512, the corresponding layer in the Decoder is a decoding layer that can process the feature map of 32*32*512 size.

[0071] In Figure 7, for the two output layers on the right side of the Encoder, the one at the top is the last residual layer ResBlock, which outputs a feature map with a size of 16*16*512, and the one at the bottom is the FC layer, which outputs style attribute information with a size of 16*512. The FC layer outputs the style attribute information to each layer of the Decoder, making it easy for the Decoder to perform full-map animation processing based on the style attribute information.

[0072] In step 650, the decoder is employed to take the sample cartoon drawings in the sample set as training targets, and the weights of the second generative model are employed as initial weights. A pre-set loss function is employed to train on the feature drawings and the style attribute information to obtain a cartoonization model.

[0073] In one embodiment, the structure of the decoder Decoder is the same as the structure of the synthesis network of the second generative model StyleGAN2 model, and training is performed using the weights of the second generative model as initial weights.

[0074] As shown in FIG. 7, each decoding layer of the Decoder receives the feature map and style attribute information input by the Encoder, decodes and synthesizes the feature map and style attribute information, and outputs the decoded result to the next layer. In this manner, the last decoding layer outputs the full map cartoon result.

[0075] In one embodiment, the loss function used to train the cartoon model is the adversarial network loss function GAN loss , the perceptual loss function loss and the regression loss function L1 loss It is possible to include a combination of loss functions, i.e.

number

[0076] Here, the adversarial network loss function GAN loss is a kind of classification loss function, which is used to judge the authenticity of the full-picture cartoon image generated by the cartoonization model, and calculate the loss according to the judgment result, so that the full-picture cartoon image generated by the cartoonization model has a more realistic cartoon feel.

[0077] In one embodiment, the following formula is employed to implement the GAN: loss Calculate:

number

[0078] Here, D represents the classifier, E represents the average value, and G(x) represents the full cartoonized image output by the cartoonization model.

[0079] Perceptual loss function loss is used to input the full-map cartoonized image output by the cartoonization model and the corresponding sample cartoon images in the sample set into a pre-installed neural network model, respectively, to obtain corresponding first and second feature images output by the neural network model, and calculate the L2 loss (least squared error) between the first feature image and the second feature image.

[0080] Illustratively, the pre-installed neural network model may be a VGG model, such as VGG-19 or VGG-16.

[0081] In one embodiment, the following formula is employed to calculate the perceptual loss can be calculated:

number

[0082] Here, E is the average value, G(x) is the full cartoon image output by the cartoonization model, and x is the sample cartoon corresponding to the sample original drawing input to the cartoonization model.

[0083] L1 loss is used to calculate the L1 loss between the full-figure cartoon image output by the cartoonization model and the corresponding sample cartoon figures in the sample set, which can be shown by adopting the following formula:

number

[0084] In addition, in the design of the loss function in this embodiment, in addition to the combination of the above three types of loss functions, other loss functions can be adopted according to the actual optimization goal, and this embodiment is not limited to this.

[0085] In this embodiment, the initial cartoon model includes an encoder and a decoder. When fitting to the initial cartoon model, the initial weights of the encoder are the weights of the encoder that has previously edited a variety of real human images, and the initial weights of the decoder are the weights of the second generative model. By adopting the above model architecture, the paired data formed from the sample real image and the sample cartoon image is the training data, and the adversarial network loss function GAN loss , the perceptual loss function loss and the regression loss function L1 lossBy combining these three loss functions and fitting a cartoonization model, the fitted cartoonization model can better extract the image feature map and sample attribute information through the encoder, and the decoder performs full-map cartoonization processing on the feature map and sample attribute information, so that the full-map cartoonized image output by the cartoonization model has a stronger cartoon feel, and the content after full-map cartoonization is more consistent with the real drawing, which further improves the robustness and generalization ability of the cartoonization model and makes it applicable to low-quality images and complex scenes.

[0086] 8 is a flow chart of a method for constructing a cartoon model according to another embodiment of the present application, which is based on the above embodiment and describes the inference process of the cartoon model in more detail. As shown in FIG. 8, the present embodiment can include the following steps:

[0087] In step 810, a pre-trained first generative model is employed to generate a preset number of sample diagrams.

[0088] In step 820, a second generative model is constructed based on the first generative model, and the second generative model is employed to generate sample cartoon drawings corresponding to each sample true drawing.

[0089] In step 830, the sample true image and the corresponding sample cartoon image are combined into a sample image pair.

[0090] In step 840, based on a sample set consisting of a plurality of the sample image pairs, weights corresponding to the second generative model are used as initial weights, and fitting is performed to a pre-established initial cartoon model to generate a cartoon model.

[0091] In step 850, a target image is obtained and the target image is input into the cartoonization model.

[0092] In one example, the target image can include an image input through an image editing page. For example, in an image editing application or an application having an image editing function, after opening an image editing page, an image introduced by a user is taken as the target image, and when the user triggers the full-picture cartooning function on the image editing page, the full-picture cartooning technology of the present application can be used to perform full-picture cartooning processing of the image.

[0093] In another example, the target image can further include multiple image frames in the target video. For example, in a live broadcast scene, if a user triggers a full-picture cartooning function in the live broadcast interface, the full-picture cartooning technology of the present application can be used to perform full-picture cartooning processing on multiple image frames in the live broadcast video. Also, in a short video or video playback scene, if a user triggers a full-picture cartooning function in the playback interface, the full-picture cartooning technology of the present application can be used to perform full-picture cartooning processing on multiple image frames in the video.

[0094] In step 860, in the cartoonization model, the encoder performs feature extraction on the target image to extract a target feature map and target style attribute information of the target image, inputs the target feature map and the target style attribute information to the decoder, and the decoder generates a corresponding full-picture cartoonized image based on the target feature map and the target style attribute information, and outputs the full-picture cartoonized image.

[0095] In this embodiment, after the cartoon model obtains a target image through the input layer of the encoder, the input layer inputs the target image to the first residual layer of the encoder as shown in Figure 7, the first residual layer extracts the feature map of the target image, inputs it to the next residual layer, and then inputs it to the corresponding layer of the decoder, and then the next residual layer continues to perform feature extraction, and in this manner, until the last residual layer and the FC layer complete the processing, the operation of the encoder is completed. Then, the decoder operates, and in each layer of the decoder, cartoonization processing is performed according to the received target feature map and target style attribute information, and the processing result is transferred to the next layer for processing, and in this manner, the last decoding layer outputs a full-picture cartoonized image to the output layer, and until the output layer outputs the full-picture cartoonized image, the operation of the decoder is completed. Then, the next target image can be processed.

[0096] In this embodiment, the full-picture cartoonization technology is realized by using the cartoon model encoder and decoder, which does not change the style of the true picture, while at the same time, has a strong full-picture cartoon style, a realistic cartoon feeling, a high sense of immersion, and is applicable to a variety of different cartoon styles.

[0097] FIG. 9 is a structural block diagram of an embodiment of the apparatus for constructing cartoon models provided in the present application, which may include the following modules: The sample true diagram generation module 910 is configured to generate a preset number of sample true diagrams by employing a first pre-trained generative model; 2nd generation Model a construction module 920 configured to construct a second generative model based on the first generative model; the sample cartoon generation module 930 is configured to employ the second generative model to generate a sample cartoon corresponding to each sample true drawing; an image pairing module 940 configured to pair the sample true illustration and a corresponding sample cartoon illustration into a sample image pair; The cartoon model fitting module 950 is configured to perform fitting to a pre-installed initial cartoon model using weights corresponding to the second generative model as initial weights based on a sample set consisting of a plurality of the sample image pairs, thereby generating a cartoon model for converting the target image into a full-picture cartoon image.

[0098] In one embodiment, the second generation Model The construction module 920 is configured as follows: adjusting the weights of the first generative model to generate an intermediate cartoon model; Weights corresponding to a specified portion of layers in the intermediate cartoon model are replaced with weights corresponding to the specified portion of layers in the first generative model, and weight interpolation is performed to generate a second generative model.

[0099] In one embodiment, the portion of the designated layers includes at least one of a layer for controlling a person's posture and a layer for controlling a person's skin color.

[0100] In one embodiment, the initial cartoon model includes an encoder and a decoder; The cartoon model fitting module 950 may include sub-modules such as: the encoding sub-module is configured to employ the encoder to perform feature extraction on sample images in the sample set to obtain corresponding feature images and style attribute information, and output the feature images and the style attribute information to the decoder; The decoding sub-module is configured to employ the decoder to take the sample cartoon drawings in the sample set as training targets, use the weights of the second generative model as initial weights, and employ a pre-set loss function to train on the feature drawings and the style attribute information to obtain a cartoonization model.

[0101] In one embodiment, the loss function includes a combination of an adversarial network loss function, a perceptual loss function, and a regression loss function L1_loss; The adversarial network loss function is used to judge the truthfulness of the full-picture cartoon image generated by the cartoonization model and calculate a loss according to the judgment result; The perceptual loss function is used to input the full-picture cartoonized image output by the cartoonization model and the corresponding sample cartoon images in the sample set into a pre-established neural network model, respectively, to obtain corresponding first and second feature images output by the neural network model, and to calculate an L2 loss between the first and second feature images; The L1_loss is used to calculate the L1 loss between the full-figure cartoon image output by the cartoonization model and the corresponding sample cartoon figures in the sample set.

[0102] In one embodiment, the encoder structure is as follows: An input layer, several residual layers and a fully connected layer, where the multiple residual layers are used for extracting feature maps in the sample true maps and outputting the feature maps to corresponding layers of a decoder, and the fully connected layer is used for extracting style attribute information of the sample true maps and outputting the style attribute information to multiple layers of a decoder.

[0103] In one embodiment, the initial weights of the encoder are weights of an encoder that has previously been edited using multiple real human images.

[0104] In one embodiment, the second generative model is a StyleGAN2 model, and the structure of the decoder is the same as the structure of the synthesis network of the StyleGAN2 model.

[0105] In one embodiment, the device may further include the following modules: a target image acquisition module configured to acquire a target image and input the target image into the cartoonization model; The full-figure cartoonization processing sub-module is configured in the cartoonization model so that the encoder performs feature extraction on the target image to extract a target feature map and target style attribute information of the target image, inputs the target feature map and the target style attribute information to the decoder, and the decoder generates a corresponding full-figure cartoonized image based on the target feature map and the target style attribute information, and outputs the full-figure cartoonized image.

[0106] In one embodiment, the target image is: Images entered via the image editing page, A plurality of image frames in the target video.

[0107] In one embodiment, the device may further include the following modules: A data augmentation module performs data augmentation on the sample set before adopting the sample set to perform model fitting, where the data augmentation is performed randomly on the sample true drawings and the sample cartoon drawings. M The configuration may include performing at least one of the following: rotate, random crop, random enlarge, random shrink.

[0108] Provided in the Examples of the present application Build a cartoon model The device is the same as in the previous embodiment of the present application. Build a cartoon model The method can be performed and has corresponding functional modules for performing the method and beneficial effects.

[0109] FIG. 10 is a structural schematic diagram of an electronic device provided in an embodiment of the present application. As shown in FIG. 10, the electronic device includes a processor 1010, a memory 1020, an input device 1030 and an output device 1040. The number of processors 1010 in the electronic device may be one or more. FIG. 10 takes one processor 1010 as an example. The processor 1010, the memory 1020, the input device 1030 and the output device 1040 in the electronic device may be connected by a bus or other manner. FIG. 10 takes the connection by a bus as an example.

[0110] The memory 1020 can be used as a computer-readable storage medium to store software programs, computer-executable programs, and modules (e.g., program instructions / modules corresponding to any of the above embodiments of the present application). The processor 1010 executes the software programs, instructions, and modules stored in the memory 1020 to perform various functional applications and data processing of the electronic device, i.e., to realize the methods described in any of the above method embodiments.

[0111] The memory 1020 may mainly include a program storage area and a data storage area, where the program storage area may store an operating system, an application required for at least one function, and the data storage area may store data created according to the use of the terminal, etc. Furthermore, the memory 1020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage component, a flash memory component, or other non-volatile solid-state storage component. In some examples, the memory 1020 may include a memory that is installed remotely with respect to the processor 1010, and these remote memories may be connected to the device / terminal / server by a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0112] The input device 1030 may be configured to receive input numeric or textual information and generate key signal inputs for user configuration and function control of the electronic device. The output device 1040 may include a display device such as a display.

[0113] An embodiment of the present application also provides a storage medium including computer-executable instructions that, when executed by a computer processor, are used to perform the method of any of the method embodiments described above. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0114] Of course, a storage medium containing computer-executable instructions provided in any embodiment of the present application is not limited to the computer-executable instructions operating in the manner described above, but may also perform related operations in the manner provided in any embodiment of the present application.

[0115] An embodiment of the present application provides another computer program product, the computer program product including computer-executable instructions, which, when executed by a computer processor, are used to perform the method of any of the embodiments described above.

[0116] Of course, the computer program product provided in the embodiments of the present application is not limited to having its computer-executable instructions operate in the manner described above, and may also perform related operations in the manner provided in any embodiment of the present application.

[0117] From the above description of the implementation mode, a person skilled in the art can clearly understand that the present application can be realized by software and necessary general-purpose hardware, and of course, can also be realized by hardware. Based on this understanding, the technical solution of the present application, essentially or contributing to the relevant technology, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, and contains several instructions to make an electronic device (which may be a personal computer, a server, or a network device, etc.) perform the above-mentioned methods of the multiple embodiments of the present application.

[0118] In addition, in the above embodiment of the device, the multiple units and modules included are divided only according to functional logic, but are not limited to the above division, as long as they realize the appropriate functions, and the specific names of the multiple functional units are also intended to make them easy to distinguish from each other, and are not intended to limit the scope of protection of the present application.

Claims

1. Employing a pre-trained first generative model to generate a preset number of sample true drawings; constructing a second generative model based on the first generative model, and employing the second generative model to generate sample cartoon drawings corresponding to each sample true drawing; combining each of the sample true images and a corresponding sample cartoon image into a sample image pair; Based on a sample set consisting of a plurality of the sample image pairs, a weight corresponding to the second generative model is set as an initial weight, and fitting is performed to an initial cartoon model set in advance to generate a cartoon model for converting the target image into a full-picture cartoon image; Including, Constructing a second generative model based on the first generative model includes: adjusting weights of the first generative model to generate an intermediate cartoon model; replacing weights corresponding to a portion of a designated layer in the intermediate cartoon model with weights corresponding to the portion of the designated layer in the first generative model, and performing weight interpolation to generate a second generative model; Including, The part of the designated layers includes at least one of a layer that controls a person's posture and a layer that controls a person's skin color. How to build a cartoon model.

2. the initial cartoon model includes an encoder and a decoder; Based on a sample set consisting of a plurality of the sample image pairs, weights corresponding to the second generative model are set as initial weights, and fitting is performed to an initial cartoon model set in advance to generate a cartoon model for converting a target image into a full-picture cartoon image; employing the encoder to perform feature extraction on sample true images in the sample set to obtain corresponding feature images and style attribute information, and outputting the feature images and the style attribute information to the decoder; The decoder is adapted to use the sample cartoon drawings in the sample set as training targets, and the weights of the second generative model are used as initial weights. A pre-defined loss function is adapted to train the feature drawings and the style attribute information to obtain a cartoon model. Including, The method of claim 1.

3. The loss function includes a combination of the following loss functions: an adversarial network loss function, a perceptual loss function, and a regression loss function L1_loss; The adversarial network loss function is used to judge the truthfulness of the full-picture cartoon image generated by the cartoonization model and calculate a loss according to the judgment result; The perceptual loss function is used to input the full-picture cartoonized image output by the cartoonization model and the corresponding sample cartoon images in the sample set into a pre-established neural network model, respectively, to obtain corresponding first and second feature images output by the neural network model, and to calculate an L2 loss between the first and second feature images; The regression loss function L1_loss is used to calculate the L1 loss between the full-figure cartoon image output by the cartoonization model and the corresponding sample cartoon figures in the sample set. The method of claim 2.

4. The structure of the encoder includes an input layer, multiple residual layers, and a fully connected layer, where the multiple residual layers are used to extract feature maps in the sample maps and output the feature maps to corresponding layers of a decoder, and the fully connected layer is used to extract style attribute information of the sample maps and output the style attribute information to multiple layers of a decoder. The method of claim 2.

5. The initial weights of the encoder are weights of an encoder that has previously edited various real human images; The method according to claim 4.

6. The second generative model is a StyleGAN2 model, and the structure of the decoder is the same as the structure of the synthesis network of the StyleGAN2 model; The method of claim 2.

7. acquiring a target image and inputting the target image into the cartoonization model; In the cartoonization model, the encoder performs feature extraction on the target image to extract a target feature map and target style attribute information of the target image, inputs the target feature map and the target style attribute information to the decoder, and the decoder generates a corresponding full-picture cartoonized image according to the target feature map and the target style attribute information, and outputs the full-picture cartoonized image; Further comprising: The method of claim 2.

8. The target image includes at least one of an image input via an image editing page and a plurality of image frames in a target video. The method of claim 7.

9. Before adopting the sample set to perform model fitting, data augmentation is performed on the sample set, where the data augmentation includes randomly performing at least one of random rotation, random cropping, random enlargement, and random shrinkage on the sample true drawing and the sample cartoon drawing; Further comprising: The method of claim 1.

10. The method includes: a sample true image generating module; a second generative model building module; a sample cartoon image generating module; an image pairing module; and a cartoon model fitting module; The sample true drawing generation module is configured to generate a preset number of sample true drawings by employing a first generative model that is pre-trained; the second generative model construction module is configured to construct a second generative model based on the first generative model; the sample cartoon diagram generation module is configured to employ the second generative model to generate a sample cartoon diagram corresponding to each sample true diagram; the image pairing module is configured to pair each of the sample true images with a corresponding sample cartoon image into a sample image pair; the cartoon model fitting module is configured to: use weights corresponding to the second generative model as initial weights based on a sample set composed of a plurality of the sample image pairs, and perform fitting to a pre-established initial cartoon model to generate a cartoon model for converting a target image into a full-picture cartoon image; the second generative model construction module is configured to adjust the weights of the first generative model to generate an intermediate cartoon model, replace weights corresponding to a portion of a designated layer in the intermediate cartoon model with weights corresponding to the portion of the designated layer in the first generative model, and perform weight interpolation to generate a second generative model; The part of the designated layers includes at least one of a layer that controls a person's posture and a layer that controls a person's skin color. A device for constructing cartoonized models.

11. one or more processors and a storage device; the storage device is configured to store one or more programs; said one or more programs, when executed by said one or more processors, causing said one or more processors to implement the method according to any one of claims 1 to 9; Electronic device for building cartoonized models.

12. A computer program is stored in the The computer program, when executed by a processor, implements the method according to any one of claims 1 to 9. A computer-readable storage medium.

13. comprising computer executable instructions, The computer executable instructions, when executed, are configured to implement the method according to any one of claims 1 to 9. Computer program.

Images