Method and system for generating cartoons from portrait photos

Abstract

The present application relates to a method and system for generating comics from portrait photos, wherein the method for generating comics from portrait photos comprises: performing feature decomposition on a portrait image and a comic style image respectively through an encoder to obtain portrait features and comic features; then performing feature normalization on the portrait features and comic features to obtain portrait normalization features and comic normalization features; then sampling and extending the portrait semantic segmentation image corresponding to the portrait image and the comic style semantic segmentation image corresponding to the comic style image to obtain a portrait extended segmentation image and a comic style extended segmentation image, and calculating the similarity matrix between the portrait normalization features and the comic normalization features, and recombining the comic normalization features through the similarity matrix to generate portrait reconstruction features; finally decoding the portrait reconstruction features to generate a portrait comic image, and completing the prediction of the portrait comic generation model. Through the present application, the accuracy of texture matching and the effect of image generation are improved, and the development cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and system for generating caricatures from portrait photos. Background Art

[0002] With the rapid development of the comics industry, it has become a popular form of entertainment loved by people from all walks of life. In the field of image processing, an ordinary portrait photo is converted into another comic-style image without changing the content. It has both the style of the comics and the recognition of the original character. This kind of comic-style application of portrait photos has received more and more attention in the field of consumer electronics and has great practical application value.

[0003] In the related technologies, the methods for generating comics from portrait photos include texture synthesis, style transfer, and deep adversarial neural networks. Texture synthesis technology has deficiencies in the semantic matching of texture features, and is prone to mismatching and generating incorrect textures, resulting in unstable generation quality; style transfer technology cannot specifically process portrait textures, and is particularly prone to producing unpleasant flaws in the facial area; deep adversarial neural network technology can generate high-quality comic images, but requires more comic materials of the same style as samples, which results in high development costs. In addition, there are related technologies that use facial feature points to draw curves of certain parts of the face, such as eyebrows and hair, and use various artificially designed features in the database to match pre-drawn comic templates, and then generate comics in a synthetic manner, which is relatively inefficient.

[0004] At present, in the related technology, when generating cartoons from portrait photos, there are problems such as insufficient matching of cartoon texture features, unstable portrait generation quality and high development cost, and no effective solution has been proposed. Summary of the invention

[0005] The embodiments of the present application provide a method and system for generating comics from portrait photos, which at least solve the problems of insufficient matching of comic texture features, unstable portrait generation quality and high development cost that may arise in the process of generating comics from portrait photos in the related art.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a comic strip from a portrait photo, the method comprising:

[0007] The encoder is used to decompose the portrait image and the comic style image respectively to obtain the portrait features and comic features;

[0008] Normalizing the portrait feature and the comic feature to obtain a portrait normalized feature and a comic normalized feature;

[0009] Sampling and extending the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image to obtain a portrait extended segmentation map and a comic style extended segmentation map, and calculating a similarity matrix between the portrait normalized features and the comic normalized features;

[0010] Recombining the cartoon normalized features through the similarity matrix to generate portrait reconstruction features;

[0011] The portrait reconstruction features are decoded to generate a portrait cartoon image, thereby completing the prediction of the portrait cartoon generation model.

[0012] In some embodiments, sampling and expanding the portrait semantic segmentation map and the comic style semantic segmentation map includes:

[0013] The resolution of the portrait semantic segmentation map is made the same as the resolution of the portrait normalization feature, and the resolution of the comic style semantic segmentation map is made the same as the resolution of the comic normalization feature;

[0014] The categorical attribute of each pixel location is represented by a one-hot vector.

[0015] In some embodiments, the similarity matrix S between the normalized portrait features and the normalized cartoon features is calculated. IC :

[0016] Calculate the distance matrix L between the normalized features of the portrait and the normalized features of the comics IC :

[0017]

[0018] in, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With S IC Same size, EX I It is the portrait extension segmentation map, EX c It is a comic style extended segmentation map, N I is the normalized feature, N C It is a normalized feature of comics;

[0019] Calculate S IC Matrix row vector

[0020]

[0021] in, That is S IC The row vector of YesL IC The row vector of the matrix N I The vector of position x and N C The distances of all vectors in the matrix.

[0022] In the embodiment, since the parameters in the model are initially unknown, it is necessary to obtain converged model parameters through training and learning before the portrait caricature generation prediction can be performed. Therefore, before the prediction of the portrait caricature generation model is performed, the method includes:

[0023] The portrait comic generation model is trained by using portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters.

[0024] In some embodiments, the training of the portrait caricature generation model to obtain network parameters includes:

[0025] The model is trained by a stochastic gradient descent algorithm to obtain the network parameters.

[0026] In a second aspect, an embodiment of the present application provides a system for generating comic strips from portrait photos, the system comprising:

[0027] An encoding module, used for performing feature decomposition on the portrait image and the comic style image through an encoder to obtain portrait features and comic features;

[0028] A feature normalization module, used to perform feature normalization on the portrait feature and the comic feature to obtain a portrait normalized feature and a comic normalized feature;

[0029] A similarity calculation module is used to sample and expand the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image to obtain a portrait extended segmentation map and a comic style extended segmentation map, and calculate a similarity matrix between the portrait normalized features and the comic normalized features;

[0030] A feature reconstruction module, used for recombining the cartoon normalization features through the similarity matrix to generate portrait reconstruction features;

[0031] The decoding module is used to decode the portrait reconstruction features, generate a portrait cartoon image, and complete the prediction of the portrait cartoon generation model.

[0032] In some embodiments, the similarity calculation module is further configured to make the resolution of the portrait semantic segmentation map the same as the resolution of the portrait normalization feature, and the resolution of the comic style semantic segmentation map the same as the resolution of the comic normalization feature.

[0033] The categorical attribute of each pixel location is represented by a one-hot vector.

[0034] In some embodiments, the similarity calculation module is further used to calculate the distance matrix L between the normalized features of the portrait and the normalized features of the comics. IC :

[0035]

[0036] in, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With SI C Same size, EX I It is the portrait extension segmentation map, EX c It is a comic style extended segmentation map, N I is the normalized feature, N C It is a normalized feature of comics;

[0037] Calculate S IC Matrix row vector

[0038]

[0039] in, That is S IC The row vector of YesL IC The row vector of the matrix N I The vector of position x and N C The distances of all vectors in the matrix.

[0040] The system also includes a model training module, before performing the prediction of the portrait caricature generation model,

[0041] The model training module is used to train the portrait comic generation model through portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters.

[0042] In some of the embodiments, the model training module is also used to train the model through a stochastic gradient descent algorithm to obtain the network parameters.

[0043] Compared with the related art, the embodiment of the present application provides a method for generating comics from portrait photos. The encoder performs feature decomposition on the portrait image and the comic style image respectively to obtain portrait features and comic features; then the portrait features and comic features are normalized to obtain portrait normalized features and comic normalized features; then the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image are sampled and expanded to obtain portrait extended segmentation map and comic style extended segmentation map, and the similarity matrix between the portrait normalized features and the comic normalized features is calculated, and the comic normalized features are recombined through the similarity matrix to generate portrait reconstruction features; finally, the portrait reconstruction features are decoded to generate a portrait comic image, and the prediction of the portrait comic generation model is completed, which solves the problems of insufficient comic texture feature matching, unstable portrait generation quality and high development cost when generating comics from portrait photos in the prior art, effectively improves the portrait comic texture matching accuracy and the portrait comic image generation quality, and reduces the development cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 is a schematic diagram of an application environment of a method for generating comic strips from portrait photos according to an embodiment of the present application;

[0046] Figure 2 is a flowchart of a method for generating comic strips from portrait photos according to an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of semantic segmentation according to an embodiment of the present application;

[0048] Figure 4 is a schematic diagram of a portrait comics image generated according to an embodiment of the present application;

[0049] Figure 5 is a structural block diagram of a portrait photo comic generation system according to an embodiment of the present application;

[0050] Figure 6 is another structural block diagram of the system for generating comic strips from portrait photos according to an embodiment of the present application;

[0051] Figure 7 is an algorithm flow chart of a method for generating comic strips from portrait photos according to an embodiment of the present application;

[0052] Figure 8It is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0054] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] The method for generating cartoons from portrait photos provided in this application can be applied to Figure 1 In the application environment shown, Figure 1 is a schematic diagram of an application environment of a method for generating comic strips from portrait photos according to an embodiment of the present application, such as Figure 1 As shown, the system of the application environment includes a server 10 and an intelligent terminal device 11, and the specific implementation method is: input a picture into the server 10, wherein the input picture includes: a portrait picture and a corresponding portrait semantic segmentation map, a comic style picture and a corresponding comic style semantic segmentation map, and the encoder is used to perform feature decomposition on the portrait picture and the comic style picture respectively to obtain portrait features and comic features; then the obtained portrait features and comic features are feature normalized to obtain portrait normalized features and comic normalized features; then the portrait semantic segmentation map corresponding to the portrait picture and the comic style semantic segmentation map corresponding to the comic style picture are sampled and expanded to obtain a portrait The image extended segmentation map and the comic style extended segmentation map are obtained, and the similarity matrix between the portrait normalized features and the comic normalized features is calculated, and the comic normalized features are recombined through the similarity matrix to generate portrait reconstruction features; finally, the obtained portrait reconstruction features are decoded to generate a portrait comic picture, which is displayed on the smart terminal device 11, completing the prediction of the entire portrait comic generation model, solving the problems of insufficient matching of comic texture features, unstable portrait generation quality and high development cost when generating comics from portrait photos in the prior art, effectively improving the portrait comic texture matching accuracy and the portrait comic picture generation quality, and reducing the development cost.

[0057] This embodiment provides a method for generating cartoons from portrait photos. Figure 2 is a flowchart of a method for generating a portrait photo comic according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0058] Step S201, feature decomposition is performed on the portrait image and the comic style image by an encoder to obtain portrait features and comic features, wherein the portrait image and the comic style image are both RGB three-channel images, and the encoder is composed of a multi-layer convolutional neural network with reduced resolution, such as VGG and RESNET convolutional networks. Optionally, in this embodiment, the encoder E is used to feature decompose the portrait image I and the comic style image C to obtain the portrait feature F I and comic feature F C , where the characteristic decomposition formula is shown in the following formula (1):

[0059] F I =E(I),F C =E(C) (1)

[0060] Compared with the traditional comic texture synthesis technology, this embodiment uses a deep neural network for encoding, which can not only extract the shallow texture of the image, but also extract the high-level semantic features in the image, effectively improving the accuracy and generation quality of texture matching;

[0061] Step S202, normalize the portrait features and the comic features to obtain normalized portrait features and normalized comic features. I and comic feature F C They may belong to different feature spaces, and the two cannot be directly compared. They need to be converted to a unified feature space through a feature normalization module N. The feature normalization module N is composed of a two-layer convolutional network. I and comic feature F C Mapped to the same comparable space, we obtain the normalized features N of the portraits. I and the comic normalization feature N C , as shown in the following formula (2):

[0062] N I =N(F I ),N C =N(F C ) (2)

[0063] Step S203, sampling and extending the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image to obtain the portrait extended segmentation map and the comic style extended segmentation map, and calculating the similarity matrix between the portrait normalized features and the comic normalized features, wherein the semantic segmentation map can be manually annotated or obtained through a semantic segmentation neural network. Figure 3 is a schematic diagram of semantic segmentation according to an embodiment of the present application, such as Figure 3 As shown, the semantic segmentation internal area map is divided into 8 categories, namely: hair, eyebrows, eyes, nose, skin, mouth, clothes, and background. Optionally, in this embodiment, the semantic segmentation map is first sampled and expanded to obtain the portrait extended segmentation map EX I and comic style extended split map EX C , and then calculate the normalized portrait feature N I and comic normalized feature N C The similarity matrix S between IC , where the normalized portrait feature N I and comic normalized feature N C is a three-dimensional feature map with dimensions w I *h I *c I and w C *h C *c C, where w and h are N I and N C The width and height of the feature matrix, c is N I and N C The width of the matrix element vector in N I and N C The c of the two matrices is the same. In addition, the similarity matrix S IC Used to measure N I and N C The similarity between them, similarity matrix S IC The size is h S *w S , where h S =w I *h I ,w S =w C *h C This embodiment uses the semantic segmentation map to calculate the similarity matrix, which can effectively improve the accuracy of comic texture matching and the quality of the generated picture;

[0064] Step S204, recombining the normalized features of the comics through a similarity matrix to generate a portrait reconstruction feature. Optionally, in this embodiment, the similarity matrix S IC Normalize the comic feature N C Recombine and reconstruct portrait features to generate portrait reconstruction feature matrix RN I , as shown in the following formula (3):

[0065] RN I =S IC *row2col(N C ) (3)

[0066] Among them, row2col() refers to the matrix N C All element vectors in are arranged in a row. In addition, the obtained portrait reconstruction feature matrix RN I The size and normalized features of the portrait N I same.

[0067] Compared with the common style transfer method, this embodiment performs semantic segmentation on the image area and reconstructs the portrait features using only the comic features of the corresponding semantic area, thus avoiding the defects caused by different semantic textures on the face of the portrait;

[0068] Step S205, decoding the portrait reconstruction features, generating a portrait cartoon image, and completing the prediction of the portrait cartoon generation model. Figure 4 is a schematic diagram of a portrait cartoon image generated according to an embodiment of the present application, such as Figure 4 As shown, optionally, this embodiment uses a cascade of several convolutional layers to reconstruct the input portrait feature RNI After convolution and upsampling, an RGB portrait cartoon image DI with the same resolution as the portrait image is output, as shown in the following formula (4):

[0069] DI=D(RN I ) (4)

[0070] This embodiment uses a deep neural network to decode the portrait reconstruction features, which can effectively improve the accuracy of the output image.

[0071] Through the above steps S201 to S205, compared with the prior art, the texture synthesis technology is prone to mismatching in the semantic matching of texture features and generates erroneous textures. The style transfer technology cannot specifically process portrait textures, especially in the facial area, which is prone to produce unpleasant flaws. The deep adversarial neural network technology requires more comic materials of the same style as samples, and there is a problem of high development cost. In response to the above problems, this embodiment uses deep learning to improve the texture synthesis method, and inputs the image into the server 10, wherein the input image includes: a portrait image and a corresponding portrait semantic segmentation map, a comic style image and a corresponding comic style semantic segmentation map, and the encoder performs feature decomposition on the portrait image and the comic style image respectively to obtain portrait features and comic features; then the obtained portrait features and comic features are feature normalized to obtain portrait normalized features and comic normalized features; then the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image are sampled and expanded to obtain a portrait extended segmentation map and a comic style extended segmentation map, and the portrait normalized segmentation map is calculated. A similarity matrix is ​​obtained between the normalized features and the comic normalized features, and the comic normalized features are recombined through the similarity matrix to generate portrait reconstruction features; finally, the obtained portrait reconstruction features are decoded to generate a portrait comic image, which is displayed on the smart terminal device 11, completing the prediction of the entire portrait comic generation model, solving the problems of insufficient comic texture feature matching, unstable portrait generation quality and high development cost when generating comics from portrait photos in the prior art, improving the accuracy of comic texture matching and the effect of portrait comic image generation, and reducing the development cost. In addition, since this embodiment adopts an end-to-end neural network solution, it also improves the image generation efficiency.

[0072] In some of the embodiments, sampling and expanding the portrait semantic segmentation map and the comic style semantic segmentation map includes: making the resolution of the portrait semantic segmentation map the same as the resolution of the portrait normalization feature, the resolution of the comic style semantic segmentation map the same as the resolution of the comic normalization feature, and then representing the classification attribute of each pixel position by a hot unique vector. Optionally, this embodiment samples the portrait semantic segmentation map and the comic style semantic segmentation map, making the resolution of the portrait semantic segmentation map the same as the resolution of the portrait normalization feature, the resolution of the comic style semantic segmentation map the same as the resolution of the comic normalization feature, and then representing the classification attribute of each pixel position by a hot unique vector, completing the expansion of the portrait semantic segmentation map and the comic style semantic segmentation map.

[0073] In some embodiments, the similarity matrix S between the normalized features of the portrait and the normalized features of the comics is calculated. IC :

[0074] Calculate the normalized feature N of the portrait I and comic normalized feature N C The distance matrix L between IC , as shown in the following formula (5):

[0075]

[0076] in, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With S IC Same size, EX I It is the portrait extension segmentation map, EX c It is a comic style extended segmentation diagram, · represents vector inner product, and * represents scalar product;

[0077] If you expand the segmentation map and If they belong to different categories, then the inner product of the two is 0. If they belong to the same category, then the inner product is 1. This ensures that different categories do not contribute to each other and avoids matching errors.

[0078] Next, calculate S IC Matrix row vector As shown in the following formula (6):

[0079]

[0080] in, That is S IC The row vector of YesL ICThe row vector of the matrix N I The vector of position x and N C The distance of all vectors in the matrix, the softmax function ensures that the row vector The sum of all elements is 1;

[0081] So far, the similarity matrix S is completed IC Calculation.

[0082] In the embodiment, since the parameters in the model are initially unknown, it is necessary to obtain converged model parameters through training and learning before the portrait comic generation prediction can be performed. Therefore, before the prediction of the portrait comic generation model, the portrait comic generation model is trained through portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters. Among them, the comic style image used as a training sample does not need to be consistent with the comic style at the time of prediction, which reduces the sample collection cost. Optionally, the prediction process of the portrait comic generation model can be shown as follows (7):

[0083] DI IC =Style(I,IM,C,CM) (7)

[0084] Among them, DI IC It means to reconstruct a portrait image using a comic picture;

[0085] Define the content loss function L con , as shown in the following formula (8):

[0086] L con (DI,I)=∑ h ‖E h (DI)-E h (I)‖2 (8)

[0087] Among them, formula (8) calculates the L2 norm of the output portrait cartoon image and the input portrait image, E h () represents the output features of the deep network in the encoding module E;

[0088] Define the style loss function L sty , as shown in the following formula (9):

[0089]

[0090] Among them, formula (9) calculates the style distance between the output portrait comic image and the input comic style image. In order to measure more accurately, the style loss is calculated according to the portrait segmentation area. i represents the segmentation area, is the shallow feature in the encoding module, The L2 norm of the mean and variance between the two features of the portrait comic image and the comic style image is calculated;

[0091] Define the reconstruction loss L resc , as shown in the following formula (10):

[0092] L resc =α1*(‖DI II -I‖2+‖DI CC -C‖2)+α2*(∑ h ‖E h (DI II )-

[0093] E h (I) ‖2+∑ h ‖E h (DI CC )-E h (C)‖2) (10)

[0094] Among them, DI II =Style(I,IM,I,IM),DI CC = Style (C, CM, C, CM), where DI II Represents the reconstruction of the portrait image using the portrait image, DI CC Indicates the reconstruction of a comic picture using a comic picture;

[0095] Define the total loss function L total , as shown in the following formula (11):

[0096] L total =ω0*L con +ω1*L sty +ω2*L resc (11)

[0097] Among them, α1, α2, ω0, ω1, ω2 are the weight coefficients of each loss, which can be determined by experience;

[0098] Compared with the deep adversarial neural network method, when realizing a certain comic style, this embodiment does not require manual drawing or collection of a large number of samples of the same style. Only one comic picture during prediction is needed to output a portrait in the comic style, which greatly reduces the cost of sample collection.

[0099] In some embodiments, training the portrait caricature generation model to obtain network parameters includes: training the model through a stochastic gradient descent algorithm to obtain network parameters, wherein the stochastic gradient descent algorithm only randomly selects a number of samples to learn and update the model parameters each time, so the learning speed is very fast each time, and supports online update, which is mostly used for learning linear classifiers under convex loss functions such as support vector machines and logistic regression. Optionally, this embodiment trains the model through a stochastic gradient descent algorithm and makes the objective function L on the sample set total Minimize, train and solve the unknown network parameters in the algorithm to improve the classification learning accuracy of the model.

[0100] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0101] This embodiment also provides a system for generating a portrait photo comic, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the terms "module", "unit", "subunit", etc. may be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0102] Figure 5 is a structural block diagram of a portrait photo comic generation system according to an embodiment of the present application, such as Figure 5 As shown, the system includes an encoding module 51, a feature normalization module 52, a similarity calculation module 53, a feature reconstruction module 54 and a decoding module 55:

[0103] The encoding module 51 is used to perform feature decomposition on the portrait image and the comic style image through an encoder to obtain portrait features and comic features respectively; the feature normalization module 52 is used to perform feature normalization on the portrait features and the comic features to obtain portrait normalized features and comic normalized features; the similarity calculation module 53 is used to sample and expand the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style map to obtain the portrait extended segmentation map and the comic style extended segmentation map, and calculate the similarity matrix between the portrait normalized features and the comic normalized features; the feature reconstruction module 54 is used to recombine the comic normalized features through the similarity matrix to generate portrait reconstruction features; the decoding module 55 is used to decode the portrait reconstruction features to generate a portrait comic image and complete the prediction of the portrait comic generation model.

[0104] Through the above system, the encoding module 51 uses a deep neural network for encoding in contrast to the traditional comic texture synthesis technology, which can not only extract the shallow texture of the image, but also extract the high-level semantic features in the image, effectively improving the accuracy and generation quality of texture matching; the portrait feature F generated by the encoder E in the feature normalization module 52 I and comic feature F C They may belong to different feature spaces, and the two cannot be directly compared. They need to be converted to a unified feature space through a feature normalization module N. The feature normalization module N is composed of a two-layer convolutional network. I and comic feature F C Mapped to the same comparable space, we obtain the normalized features N of the portraits. I and the comic normalization feature N C ; The similarity calculation module 53 uses the semantic segmentation map to calculate the similarity matrix, which can effectively improve the accuracy of comic texture matching and the quality of the generated picture; the feature reconstruction module 54 compares the common style transfer method, performs semantic segmentation on the image area, and only uses the comic features of the corresponding semantic area to reconstruct the portrait features, avoiding the defects brought to the portrait face by different semantic textures; the decoding module 55 uses a deep neural network to decode the portrait reconstruction features, which can effectively improve the accuracy of the output picture. The whole system solves the problems of insufficient comic texture feature matching, unstable portrait generation quality and high development cost when generating comics from portrait photos in the prior art, improves the accuracy of comic texture matching and the effect of portrait comic image generation, and reduces the development cost.

[0105] In some of the embodiments, the similarity calculation module 53 is also used to make the resolution of the portrait semantic segmentation map the same as the resolution of the portrait normalization feature, the resolution of the comic style semantic segmentation map the same as the resolution of the comic normalization feature, and then use a hot unique vector to represent the classification attribute of each pixel position. Optionally, this embodiment samples the portrait semantic segmentation map and the comic style semantic segmentation map, so that the resolution of the portrait semantic segmentation map is the same as the resolution of the portrait normalization feature, the resolution of the comic style semantic segmentation map is the same as the resolution of the comic normalization feature, and then uses a hot unique vector to represent the classification attribute of each pixel position, thereby completing the expansion of the portrait semantic segmentation map and the comic style semantic segmentation map.

[0106] In some embodiments, the similarity calculation module 53 is further used to calculate the similarity matrix S between the normalized features of the portrait and the normalized features of the comics. IC :

[0107] Calculate the normalized feature N of the portrait I and comic normalized feature N C The distance matrix L between IC, as shown in the following formula (5):

[0108]

[0109] Among them, among them, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With S IC Same size, EX I It is the portrait extension segmentation map, EX c It is a comic style extended segmentation diagram, · represents vector inner product, and * represents scalar product;

[0110] If you expand the segmentation map and If they belong to different categories, then the inner product of the two is 0. If they belong to the same category, then the inner product is 1. This ensures that different categories do not contribute to each other and avoids matching errors.

[0111] Next, calculate S IC Matrix row vector As shown in the following formula (6):

[0112]

[0113] in, That is S IC The row vector of YesL IC The row vector of the matrix N I The vector of position x and N C The distance of all vectors in the matrix, the softmax function ensures that the row vector The sum of all elements is 1;

[0114] So far, the similarity matrix S is completed IC Calculation.

[0115] The system also includes a model training module 61, Figure 6 is another structural block diagram of the portrait photo comic generation system according to an embodiment of the present application, such as Figure 6 As shown, before the prediction of the portrait comic generation model is performed, the model training module trains the portrait comic generation model through portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters. Among them, the comic style image used as a training sample does not need to be consistent with the comic style during prediction, which reduces the sample collection cost. Optionally, the prediction process of the portrait comic generation model can be shown as follows (7):

[0116] DI IC =Style(I,IM,C,CM) (7)

[0117] Among them, DI IC It means to reconstruct a portrait image using a comic picture;

[0118] Define the content loss function L con , as shown in the following formula (8):

[0119] L con (DI,I)=∑ h ‖E h (DI)-E h (I)‖2 (8)

[0120] Among them, formula (8) calculates the L2 norm of the output portrait cartoon image and the input portrait image, E h () represents the output features of the deep network in the encoding module E;

[0121] Define the style loss function L sty , as shown in the following formula (9):

[0122]

[0123] Among them, formula (9) calculates the style distance between the output portrait comic image and the input comic style image. In order to measure more accurately, the style loss is calculated according to the portrait segmentation area. i represents the segmentation area, is the shallow feature in the encoding module, The L2 norm of the mean and variance between the two features of the portrait comic image and the comic style image is calculated;

[0124] Define the reconstruction loss L resc , as shown in the following formula (10):

[0125] L resc =α1*(‖DI II -I‖2+‖DI CC -C‖2)+α2*(∑ h ‖E h (DI II )-

[0126] E h (I) ‖2+∑ h ‖E h (DI CC )-E h (C)‖2) (10)

[0127] Among them, DI II=Style(I,IM,I,IM),DI CC = Style (C, CM, C, CM), where DI II Represents the reconstruction of the portrait image using the portrait image, DI CC Indicates the reconstruction of a comic picture using a comic picture;

[0128] Define the total loss function L total , as shown in the following formula (11):

[0129] L total =ω0*L con +ω1*L sty +ω2*L resc (11)

[0130] Among them, α1, α2, ω0, ω1, ω2 are the weight coefficients of each loss, which can be determined by experience;

[0131] Compared with the deep adversarial neural network method, when realizing a certain comic style, this embodiment does not require manual drawing or collection of a large number of samples of the same style. Only one comic picture during prediction is needed to output a portrait in the comic style, which greatly reduces the cost of sample collection.

[0132] In some embodiments, the model training module 61 is also used to train the model through a stochastic gradient descent algorithm to obtain network parameters, wherein the stochastic gradient descent algorithm only randomly selects a number of samples to learn and update the model parameters each time, so the learning speed is very fast each time, and supports online update, and is mostly used for learning linear classifiers under convex loss functions such as support vector machines and logistic regression. Optionally, this embodiment trains the model through a stochastic gradient descent algorithm, and uses the objective function L on the sample set. total Minimize, train and solve the unknown network parameters in the algorithm to improve the classification learning accuracy of the model.

[0133] The present invention is described in detail below in conjunction with application scenarios.

[0134] The purpose of the present invention is to provide a method and system for generating cartoons from portrait photos. Figure 7 is an algorithm flow chart of a method for generating comic strips from portrait photos according to an embodiment of the present application, such as Figure 7 As shown, the present invention starts from the traditional texture synthesis idea, uses deep learning to improve the shortcomings of the texture synthesis solution, and improves the accuracy of texture matching and the effect of image generation.

[0135] The process steps of the technical solution of the method for generating comic strips from portrait photos in this embodiment include:

[0136] S1, through the encoder E, the portrait image I and the comic style image C are decomposed to obtain the portrait feature F I and comic feature F C ;

[0137] S2, through the feature normalization module N, the portrait feature F I and comic feature F C Mapped to the same comparable space, we obtain the normalized features N of the portraits. I and the comic normalization feature N C ;

[0138] S3, sampling and expanding the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image, to obtain the portrait extended segmentation map EX I and comic style extended split map EX C , and then calculate the normalized portrait feature N I and comic normalized feature N C The similarity matrix S between IC ;

[0139] S4, the similarity matrix S calculated by the R module IC Normalize the comic feature N C Recombine and reconstruct the portrait features to generate the portrait reconstruction feature matrix RN I ;

[0140] S5, D module reconstructs the portrait feature RN I Decode and generate the RGB portrait comic image DI to complete the prediction of the portrait comic generation model.

[0141] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0142] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0143] In addition, in combination with the method for generating comic strips from portrait photos in the above embodiments, the present application embodiment may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any of the methods for generating comic strips from portrait photos in the above embodiments is implemented.

[0144] In one embodiment, Figure 8 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 8As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in Figure 8 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating a portrait photo comic is implemented.

[0145] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0147] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating cartoons from portrait photos, characterized in that: The method comprises: The encoder is used to decompose the portrait image and the comic style image respectively to obtain the portrait features and comic features; Normalizing the portrait feature and the comic feature to obtain a portrait normalized feature and a comic normalized feature; The portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image are sampled and expanded to obtain a portrait extended segmentation map and a comic style extended segmentation map, and a similarity matrix between the portrait normalized features and the comic normalized features is calculated. The similarity matrix S between the portrait normalized features and the comic normalized features is calculated. IC include: Calculate the distance matrix L between the normalized features of the portrait and the normalized features of the comics IC : in, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With S IC Same size, EX I It is the portrait extension segmentation map, EX C It is a comic style extended segmentation map, N I is the normalized feature, N C It is a normalized feature of comics; Calculate S IC Matrix row vector in, YesS IC The row vector of YesL IC The row vector of the matrix N I The vector of position x and N C The distances of all vectors in the matrix; Recombining the cartoon normalized features through the similarity matrix to generate portrait reconstruction features; The portrait reconstruction features are decoded to generate a portrait cartoon image, thereby completing the prediction of the portrait cartoon generation model.

2. The method according to claim 1, characterized in that The sampling and expansion of the portrait semantic segmentation map and the comic style semantic segmentation map includes: The resolution of the portrait semantic segmentation map is made the same as the resolution of the portrait normalization feature, and the resolution of the comic style semantic segmentation map is made the same as the resolution of the comic normalization feature; The categorical attribute of each pixel location is represented by a one-hot vector.

3. The method according to claim 1, characterized in that Before performing prediction of the portrait caricature generation model, the method includes: The portrait comic generation model is trained by using portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters.

4. The method according to claim 3, characterized in that The network parameters obtained by training the portrait comic generation model include: The model is trained by a stochastic gradient descent algorithm to obtain the network parameters.

5. A system for generating cartoons from portrait photos, characterized in that: The system comprises: An encoding module, used for performing feature decomposition on the portrait image and the comic style image through an encoder to obtain portrait features and comic features; A feature normalization module, used to perform feature normalization on the portrait feature and the comic feature to obtain a portrait normalized feature and a comic normalized feature; A similarity calculation module is used to sample and expand the portrait semantic segmentation map corresponding to the portrait image and the comic style semantic segmentation map corresponding to the comic style image to obtain the portrait extension segmentation map and the comic style extension segmentation map, and calculate the similarity matrix between the portrait normalized features and the comic normalized features. The similarity calculation module is also used to calculate the distance matrix L between the portrait normalized features and the comic normalized features. IC : in, YesL IC The specific elements in Represents the matrix N I The vector of position x and N C The distance between the vectors at the matrix position y, L IC With S IC Same size, EX I It is the portrait extension segmentation map, EX C It is a comic style extended segmentation map, N I is the normalized feature, N C It is a normalized feature of comics; Calculate S IC Matrix row vector in, YesS IC The row vector of YesL IC The row vector of the matrix N I The vector of position x and N C The distances of all vectors in the matrix; A feature reconstruction module, used for recombining the cartoon normalization features through the similarity matrix to generate portrait reconstruction features; The decoding module is used to decode the portrait reconstruction features, generate a portrait cartoon image, and complete the prediction of the portrait cartoon generation model.

6. The system according to claim 5, characterized in that The similarity calculation module is further used to make the resolution of the portrait semantic segmentation map the same as the resolution of the portrait normalization feature, and the resolution of the comic style semantic segmentation map the same as the resolution of the comic normalization feature. The categorical attribute of each pixel location is represented by a one-hot vector.

7. The system according to claim 5, characterized in that The system also includes a model training module, before performing the prediction of the portrait caricature generation model, The model training module is used to train the portrait comic generation model through portrait image training samples, comic style image training samples and corresponding portrait semantic segmentation image training samples and comic style semantic segmentation image training samples to obtain network parameters.

8. The system according to claim 7, characterized in that The model training module is also used to train the model through a stochastic gradient descent algorithm to obtain the network parameters.

Images