Generative adversarial neural network based on image creation and cartoon face image generation method
Through the generative adversarial neural network based on image creation, the cartoon image generation process is decomposed and the multi-module collaborative optimization is combined, the problem of low quality of cartoon face image generation in the existing technology is solved, and efficient and high-quality cartoon faces are achieved Image generation meets industry needs.
Patent Information
- Application Number
- CN202410939180.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
AI Technical Summary
The existing natural image style migration technology cannot quickly generate high-quality cartoon face images, and the generated cartoon face images have problems such as missing semantic content, insignificant cartoon texture, and incomplete structure.
Adopt generative adversarial neural network based on image creation, by dividing the cartoon image generation process into texture module, surface module, local texture module, content module and structural module, and using generation module, enhancement module, cartoon style extraction module and real human face The extraction modules cooperate with each other to generate cartoon face images with clear outlines, more details, soft strokes, many thick lines, and smooth surfaces.
The generation efficiency and quality of cartoon face images have been improved, and the generated images are closer to cartoon works created by artists, meeting the needs of the cartoon industry and social media, and making up for the shortcomings of the existing technology.
Smart Images

Figure CN118941659A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer graphics, including image editing, face cartoon style transfer, etc., and specifically relates to a generative adversarial neural network based on image creation and a cartoon face image generation method. Background Art
[0002] Cartoon face images are popular with people due to their unique expression and are widely used in the cartoon industry and social media. In recent years, people's aesthetic taste has gradually improved, leading to higher and higher requirements for the quality of cartoon face images. At the same time, the rapid development of the cartoon industry and the diversified development of the social media field have led to a significant increase in people's requirements for the quantity and quality of cartoon face images. However, the efficiency of artists creating cartoon face images is slow and the technical requirements of artists are high, making it difficult to complete a large number of cartoon face image generation tasks; at present, computer-generated cartoon face images are not obviously cartoonized, the content is incomplete, and the details are few, which makes it difficult to meet people's requirements for cartoon image quality. Therefore, it is urgent to find a method that can quickly generate high-quality cartoon face images. In recent years, the task of face cartoon style transfer achieved with the help of computer technology has developed rapidly. Face cartoon style transfer is a challenging computer vision task that aims to transfer the cartoon face style of Figure A to Figure B. How to use face cartoon style transfer technology to quickly generate high-quality cartoon face images is a current hot topic in digital media, machine vision and other fields and related fields.
[0003] The research on face cartoon style transfer technology not only has important theoretical research significance, but also can promote its application development in the cartoon industry, social media and other fields. In the early years, the development of image style transfer technology was relatively slow. Usually, a related algorithm could only be used for a single image, and the processing effect was not ideal and the efficiency was low. In recent years, the rapid development of deep learning technology has promoted the development of image style transfer technology. However, cartoon images have the characteristics of smooth color changes, clear contours, and obvious changes in details, which make cartoon images different from natural images. Existing style transfer methods are mainly developed for applications in natural images and cannot be well applied to the production of cartoon images.
[0004] In order to solve the problem that the existing style transfer technology cannot quickly generate high-quality cartoon face images, the present invention proposes a generative adversarial neural network based on cartoon image creation and a method for generating cartoon face images. Compared with the traditional method, the cartoon face image generated by this method has the characteristics of clear contours, more details, softer strokes, more thick lines, smooth surface, etc. The generated cartoon face image is of high quality and has obvious cartoon effect. At the same time, the generative adversarial neural network based on image creation proposed by this method requires fewer model parameters, and the time required for training and testing is shorter, so the creation efficiency of generating cartoon face images can be improved. In summary, this method can quickly and efficiently generate high-quality cartoon face images, greatly meet the needs of the cartoon industry and social media, and has good application value in image analysis, face cartoon style transfer, cartoon industry, social media and other fields. Summary of the invention
[0005] The purpose of the present invention is to overcome the problem that the existing technology for natural image style transfer cannot meet the needs of the cartoon industry, social media and other fields to produce large quantities of high-quality cartoon face images, and the cartoon face images created by the current technology have the problems of missing semantic content, insignificant cartoon texture, incomplete structure and other cartoon style problems, and provide a generative adversarial neural network based on image creation and a cartoon face image generation method. According to the process of image creation, the method divides the generation network of the cartoon image into a texture module, a surface module, a local texture module, a content module and a structure module. The cartoon face image generated by the method makes up for the defects of the existing method for natural image style transfer applied to the creation of face cartoon images and is more efficient. The generated cartoon image is closer to the cartoon work created by the artist, wherein the generated cartoon image has a clear outline, more details, softer brushstrokes, more thick lines, a smooth surface, smooth colors in the non-edge area and sparse color blocks, and the whole image looks softer.
[0006] In order to achieve the above-mentioned purpose, the present invention proposes a technical solution, a generative adversarial neural network based on image creation, characterized in that the network is constructed based on the image creation process and is used to quickly generate high-quality cartoon face images; the network contains two inputs, namely input 1 and input 2, wherein the former is the input natural face image, and the latter is the input cartoon style face image; the network is composed of a generation module, an enhancement module, a cartoon style extraction module, and a real face extraction module; the generation module contains an input and an output, the one input is the input 1, and the one output is the output G1, which is an image G1 generated according to the input 1 with desired cartoon style features including texture, surface, local texture features, etc.; the enhancement module contains two inputs and one output, the two inputs are the input 1 and the output G1, and the The output of the enhancement module is output G2, which integrates the features of the two inputs of the enhancement module, assigns the cartoon style features of the output G1 to the input 1, and obtains cartoon style image data with clearer image contours, more details, softer strokes, more thick lines, and smoother surfaces; the cartoon style extraction module contains two inputs and three implicit outputs, the two inputs are the output G2 and the input 2, the module extracts and compares and analyzes the texture, surface, and local texture features of its two inputs, and continuously optimizes and regulates the network structure and performance parameters of the generation module through implicit outputs; the real face extraction module contains two inputs and two implicit outputs, the two inputs are the output G2 and the input 1, the module extracts and compares and analyzes the content and structural features of its two inputs, and continuously optimizes and regulates the structure and performance parameters of the generation module through implicit outputs;
[0007] The generation module is composed of convolution layer A, leaky relu activation function layer, convolution layer A1, convolution layer A2, convolution layer A3, operation layer 1, convolution layer A4, operation layer 2, convolution layer A5, operation layer 3, convolution layer A6, operation layer 4, bilinear layer 1, operation layer 5, convolution layer A7, bilinear layer 2, operation layer 6, convolution layer A8, and compression layer connected in series in sequence; the operation layer 1, operation layer 2, operation layer 3, operation layer 4, operation layer 5, and operation layer 6 each have two inputs and one output, and the function realized is to perform a sum operation on the two inputs; the two inputs of the operation layer 1 are the convolution layer A2, The output of convolution layer A3, the two inputs of the operation layer 2 are the output of the operation layer 1 and the output of the convolution layer A4, the two inputs of the operation layer 3 are the output of the operation layer 2 and the output of the convolution layer A5, the two inputs of the operation layer 4 are the output of the operation layer 3 and the output of the convolution layer A6, the two inputs of the operation layer 5 are the output of the bilinear layer 1 and the output of the convolution layer A1, the two inputs of the operation layer 6 are the output of the bilinear layer 2 and the output of the activation function layer; the convolution layer A1 is composed of convolution layer B, Leaky relu activation function layer, convolution layer C, and leaky relu activation function layer are sequentially connected in series; the convolution layer A2 is composed of convolution layer D, leaky relu activation function layer, convolution layer E, and leaky relu activation function layer in series; the convolution layer A3, convolution layer A4, convolution layer A5, and convolution layer A6 are all composed of convolution layer F, leaky relu activation function layer, and convolution layer F in series; the convolution layer A7 is composed of convolution layer C and leaky relu activation function layer in series; the convolution layer A8 is composed of convolution layer F, leaky The relu activation function layer and the convolution layer G are connected in series in sequence; the functions of the convolution layers A1, A2, A3, A4, A5, A6, A7 and A8 are to extract features from their inputs and obtain deep feature information of the image; the functions of the convolution layers A, B, C, D, E, F and G are to extract features from their inputs; the function of the leaky relu activation function layer is to adjust the data whose input tensor is less than zero to 0.2 times of the original; the bilinear layer 1 is composed of the convolution layers C, Leaky The relu activation function layer and the linear layer 1 are connected in series in sequence, and the linear layer 1 is used to adjust the size of its input; the bilinear layer 2 is composed of the convolution layer F, the leakyrelu activation function layer, and the linear layer 1 in series in sequence; the bilinear layer 1 and the bilinear layer 2 have the same function, both of which are to extract features and change the size of their input tensors; the compression layer is used to adjust the input tensor data to [-1, 1];
[0008] The enhancement module contains two inputs and one output, the two inputs are the input 1 and the input G1, the input G1 corresponds to the output G1 of the generation module, and the one output is the output G2; the enhancement module is composed of a computing layer 7, a computing layer 8, a computing layer 9, a computing layer 10, a computing layer 11, a computing layer 12, a computing layer 13, a computing layer 14, a smoothing filter layer and a standardization layer; the function of the computing layer 7 is to perform a square operation on the input tensor; the function of the computing layer 8 and the computing layer 9 is to perform a multiplication operation on the two input tensors; the function of the computing layer 10, the computing layer 13, and the computing layer 14 is to perform a compound operation on the input, and the compound operation is The corresponding position data of multiple input tensors are added, subtracted, multiplied and divided; the operation layer 11 and the operation layer 12 are used to perform subtraction operations on the two inputs; the smoothing filter layer has one input and one output, and its function is to perform a smoothing operation on its input, and the smoothing operation is to take the average value of the small area data in the input tensor as the central value; the standardization layer has one input and one output, and its function is to reduce the data at all positions of the input tensor, and the reduction operation is to divide the data at all positions of the input tensor by the size of the input tensor; the function of the enhancement module is to use G1 as the guiding information to perform cartoon style enhancement on input 1, which is achieved through the following process:
[0009] Process 1: The input 1 is processed in series by the operation layer 7, the smoothing filter layer, and the normalization layer to obtain the output tensor data Q1;
[0010] Process 2: The input 1 is processed in series with the smoothing filter layer and the normalization layer to obtain the output tensor data Q2; the input G1 is processed in series with the smoothing filter layer and the normalization layer to obtain the output tensor data Q3;
[0011] Process 3: Input the obtained data Q2 and data Q3 to the operation layer 9, and obtain tensor data Q4=Q2*Q3 through multiplication operation of the operation layer 9;
[0012] Process 4: The input 1 and input G1 are input to the operation layer 8, and then sequentially processed in series by the smoothing filter layer and the normalization layer to obtain the output tensor data Q5;
[0013] Process 5: Input the obtained data Q1 and Q4 to the operation layer 11, and obtain tensor data Q6=Q1-Q4 through subtraction operation of the operation layer 11; input the obtained data Q4 and Q5 to the operation layer 12, and obtain tensor data Q7=Q5-Q4 through subtraction operation of the operation layer 12;
[0014] Process 6: Input the obtained data Q6 and data Q7 to the operation layer 10, and obtain tensor data Q8=Q7 / (Q6+0.01)=Q8 through compound operation of the operation layer 10;
[0015] Process 7: Input the obtained data Q8 to the smoothing filter layer and the standardization layer, and obtain the output tensor data Q9 through the series processing;
[0016] Process 8: Input the obtained data Q2, data Q3 and data Q8 to the operation layer 13, and obtain tensor data Q10 = Q3-Q2*Q8 through compound operation of the operation layer 13;
[0017] Process 9: Input the obtained data Q10 into the smoothing filter layer and the standardization layer, and obtain the tensor data Q11 through the series processing;
[0018] Process 10: Input the input 1 and the obtained data Q9 and data Q11 to the operation layer 14, and obtain the output tensor data G2 = the input 1*Q9+Q11 through compound operation processing, that is, the output G2;
[0019] The cartoon style extraction module is composed of a texture module, a surface module and a local texture module. Each module contains two inputs and one implicit output. The two inputs are the input 2 and the input G2. The input G2 corresponds to the output G2 of the enhancement module. The module extracts and compares and analyzes the texture, surface and local texture features of its input, and then feeds the results back to the generation module and adjusts the corresponding submodules of the module, so as to continuously optimize and regulate its network structure and performance parameters.
[0020] The texture module is composed of a color decomposition layer, a texture discriminator A, a texture discriminator B and a texture module optimization controller, which contains two inputs and one implicit output. The function of the module is to extract and compare the details and structural edge features of the grayscale images of its two inputs, back-propagate the comparison results, and continuously optimize the structure and performance parameters of the corresponding submodules of the module and the generation module; the color decomposition layer contains two inputs and two outputs, the two inputs are the input G2 and the input 2, and the two outputs are output gray_fake and output gray_cartoon, which correspond to the grayscale image of the input G2 and The grayscale image of the input 2, the function of the color decomposition layer is to convert the two color image data of its input into grayscale image data; the texture discriminator A and the texture discriminator B have the same structure and function, and both contain an input and an output; the input of the texture discriminator A is the output gray_fake, and its output is output texture_1, the input of the texture discriminator B is the output gray_cartoon, and its output is output texture_2; the functions of the texture discriminator A and the texture discriminator B are to extract the texture feature information such as the details and structural edges of their inputs, which are both composed of the convolutional layer H, the Leaky relu activation function layer, convolution layer I, leaky relu activation function layer, convolution layer J, leaky relu activation function layer, convolution layer K, leaky relu activation function layer, convolution layer L, leaky relu activation function layer, convolution layer M, leaky relu activation function layer, and convolution layer N are sequentially connected in series; the functions of the convolution layer H, convolution layer I, convolution layer J, convolution layer K, convolution layer L, convolution layer M, and convolution layer N are all to extract the details and structural edge features of their inputs; the texture module optimization controller contains two inputs and three implicit outputs, the two inputs are the output texture_1 and the output texture_2, the three implicit outputs are output texture feedback 1, output texture feedback 2 and output texture feedback 3, and the output texture feedback 3 is the implicit output of the texture module; the texture module optimization controller is composed of a loss function and an optimizer, the loss function contains two inputs and one output, and the one output is the input of the optimizer; the loss function of the texture module optimization controller is
[0021] l texture =λ1[logD(f rcs (I C ))+log(1-D(f rcs (G2)))]
[0022] Among them, f rcsis the color decomposition layer, I C , G2 are the input 2 and input G2 respectively, D is the texture discriminator A or texture discriminator B, log is the logarithmic operation, l texture is the texture loss value, λ1∈[0,10] is the weight coefficient of the texture module; the optimizer optimizes and adjusts the network structure and performance parameters of the texture discriminator A, texture discriminator B and generation module through its three implicit outputs to make the gradient of the neurons converge as a whole. The principle of optimization and adjustment is to make the loss function l texture Continuously descending and converging;
[0023] The surface module is composed of smooth image processing A, smooth image processing B, surface discriminator A, surface discriminator B and surface module optimization controller, which contains two inputs and one implicit output, and its function is to extract and compare the surface features of the pixel smooth area of the input G2 and input 2, and then back-propagate the comparison results and feed them back to the generation module and the corresponding submodules of the module to continuously optimize and regulate its network structure and performance parameters; the smooth image processing A and the smooth image processing B have the same structure and function, both contain one input and one output, and their function is to smooth and denoise their inputs to obtain smoother and clearer image data; the input of the smooth image processing A is the input G2, which corresponds to the output G2, and its output is output blur_fake, which means the blurred image of the input G2; the input of the smooth image processing B is the input 2, and its output is output blur_cartoon, Its meaning is the blurred image of the input 2; the surface discriminator A, the surface discriminator B and the texture discriminator A have exactly the same structure, and all contain an input and an output, which are surface feature image data for extracting the input while removing edges, details, and textures and retaining colors; the input of the surface discriminator A is the output blur_fake and its output is the output surface_1, the input of the surface discriminator B is the output blur_cartoon and its output is the output surface_2; the surface module optimization controller contains two inputs and three implicit outputs, the two inputs are the output blur_fake and the output blur_cartoon, the three implicit outputs are surface feedback 1, surface feedback 2 and surface feedback 3, and the surface feedback 3 is the implicit output of the surface module; the surface module optimization controller is composed of a loss function and an optimizer connected in series in sequence, and the loss function is
[0024] l surface =λ2[logD(f dgf (I C))+log(1-D(f dgf (G2))]
[0025] Among them, f dgf are the smooth image processing A and the smooth image processing B, D are the surface discriminator A and the surface discriminator B, log is the logarithmic operation, l surface is the surface loss value, λ2∈[1,2] is the weight coefficient of the surface module; the optimizer optimizes the network structure and performance parameters of the surface discriminator A, surface discriminator B and generation module through its three implicit outputs, so that the gradient of the neurons converges as a whole. The principle of optimization is to make the loss function l surface The value of is decreasing and converging;
[0026] The smooth image processing A and the smooth image processing B are both composed of a smooth filter layer, a standardization layer, a computing layer 15, a computing layer 16, a computing layer 17, a computing layer 18, a computing layer 19, and a computing layer 20; the computing layer 15 and the computing layer 16 each have one input and one output; the computing layer 17 has two inputs and one output, and implements a subtraction operation on its input; the computing layer 18 has one input and one output, the computing layer 19 has two inputs and one output, and the computing layer 20 has three inputs and one output, and the function implemented is a composite operation on its input; the smooth filter layer has one input and one output, and is used to smooth its input, and the smoothing process refers to taking the average value of the data of the small area in the input tensor as the central value; the standardization layer has one input and one output, and its function is to reduce the data at all positions of the input tensor, and the reduction operation is to divide the data at all positions of the input tensor by the size of the input tensor; the function of the smooth image processing A and the smooth image processing B is to keep the edge of the input image clear, which is achieved through the following process:
[0027] Process 1: Let the input be image data I, and process the image I in series through the smoothing filter layer, the normalization layer and the operation layer 16 to obtain output data P1;
[0028] Process 2: The data image I is processed in series by the operation layer 15, the smoothing filter layer, and the normalization layer to obtain output data P2;
[0029] Process 3: input the obtained data P1 and data P2 to the operation layer 17, and obtain output data P3=P2-P1 through subtraction operation of the operation layer 17;
[0030] Process 4: input the obtained data P3 to the operation layer 18, and obtain the output data P4=P3 / (P3+ε1) after the compound operation of the operation layer 18, wherein ε1=0.01 is a small constant used to avoid the denominator being zero;
[0031] Process 5: input the obtained data P4 and data P1 to the operation layer 19, and obtain output data P5=P1-P4*P1 through compound operation of the operation layer 19;
[0032] Process 6: input the obtained data P4 to the smoothing filter layer and the standardization layer, and then sequentially connect them in series to obtain the output data P6;
[0033] Process 7: input the obtained data P5 to the smoothing filter layer and the standardization layer, and then sequentially connect them in series to obtain the output data P7;
[0034] Process 8: The obtained data P6, data P7 and the image data I are input to the operation layer 20, and the output data P8=P6*image I+P7 is obtained through the compound operation of the operation layer 20. The data P8 is the output of the smoothing image processing A and the smoothing image processing B.
[0035] The local texture module is composed of a patch extraction layer A, a patch extraction layer B, a color decomposition layer, a local texture discriminator A, a local texture discriminator B and a local texture module optimization controller, which contains two inputs and one implicit output. The function of the module is to extract and compare the details and structural edge local texture features of the corresponding grayscale image of the small area image of its input, and to back-propagate the comparison results, and continuously optimize the structure and performance parameters of the corresponding submodules of the module and the generation module; the patch extraction layer A and the patch extraction layer B have exactly the same structure and function, both contain one input and one output, and their function is to extract the small area image data in the input G2 and the input 2 respectively; the input of the patch extraction layer A is the input G2 and its output is output G2_patches, the input of the patch extraction layer B is the input 2 and its output is cartoon_patches; the color decomposition layer has two inputs and two outputs, the two inputs are the output G2_patches and the output cartoon_patches, and the two outputs are input Output G2_patche_gray and output cartoon_patches_gray, which are the small area grayscale images of their inputs respectively; the local texture discriminator A and the local texture discriminator B have exactly the same structure and function, and both contain one input and one output; the input of the local texture discriminator A is the output G2_patche_gray and its output is output local_1, the input of the local texture discriminator B is the output cartoon_patches_gray and its output is local_2; the local texture module optimization controller contains two inputs and three implicit outputs, the two inputs are the output local_1 and the output local_2, the three implicit outputs are local texture feedback 1, local texture feedback 2 and local texture feedback 3, and the local texture feedback 3 is the implicit output of the local texture module; the local texture module optimization controller and the texture module optimization controller have exactly the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is
[0036] L local =λ3[logD(f rcs (f p (I C ))+log(1-D(f rcs (f p (G2))))]
[0037] Among them, f p are the patch extraction layer A and the patch extraction layer B, f rcsis the color decomposition layer, D is the local texture discriminator A, the local texture discriminator B, log is the logarithmic operation, L local is the local texture loss value, λ3∈[200,400] is the weight coefficient of the local texture module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the local texture discriminator A, the local texture discriminator B and the generation module respectively, so that the gradient of the neurons of the local texture discriminator A and the generation module converges as a whole. The principle of the optimization process is to make the loss function L local The value of keeps decreasing;
[0038] The patch extraction layer A and the patch extraction layer B have the same structure and function, and are both composed of a smooth image processing C, a convolution module, an operation layer 21, a connection layer, a convolution layer 5, and an operation layer 22 connected in series in sequence; the smooth image processing C has exactly the same structure and function as the smooth image processing A and the smooth image processing B, and contains one input and one output, and the one input is the input of the patch extraction layer; the convolution module contains one input and four outputs, and its function is to extract the edge information of its input tensor to obtain the edge line image feature data of the input tensor, and the one input is the output of the smooth image processing C. The convolution module is composed of a convolution layer 1, a convolution layer 2, a convolution layer 3 and a convolution layer 4 in parallel, and the convolution layer 1, the convolution layer 2, the convolution layer 3 and the convolution layer 4 all contain one input and one input, and their inputs are all inputs of the convolution module; the operation layer 21 contains four inputs and one output, The four inputs are the four outputs of the convolution module, and the function implemented is to take the absolute value of the input and add and scale it to [0, 1] to obtain the rough edge map feature data containing the edge features of the input of the patch extraction layer A and the patch extraction layer B; the connection layer contains two inputs and one output, and the two inputs are the output of the operation layer 21 and the input of the patch extraction layer A or the patch extraction layer B, and its function is to splice the two inputs in the fourth dimension to obtain the edge map and the image data of the corresponding positions of the input of the patch extraction layer A or the patch extraction layer B overlapping each other; the convolution layer 5 contains one input and one output, and its function is to extract the small area image of the input tensor, and the function of the operation layer 22 is to calculate the pixel values of the small area image of the convolution layer 5 and sort them in descending order; the local texture discriminator A and the local texture discriminator B are both composed of the convolution layer S, the normalization layer, the Leaky relu activation function layer, convolution layer T, normalization layer, leaky relu activation function layer, convolution layer U, normalization layer, leaky relu activation function layer, convolution layer V, normalization layer, leaky relu activation function layer, and convolution layer W are sequentially connected in series, and their function is to extract and analyze the local texture features of their input, and obtain the most significant details and structural edge image feature data of the input image; the functions of the convolution layer S, convolution layer T, convolution layer U, convolution layer V, and convolution layer W are all to extract the local texture features of their input tensors, and the function of the normalization layer is to make the mean and variance of each channel of the input tensor 0 and 1 respectively;
[0039] The real face extraction module is composed of a content module and a structure module; the content module contains two inputs and one implicit output, the inputs are the input 1 and input G2, and the input G2 corresponds to the output G2; the content module is composed of a content discriminator A, a content discriminator B and a content module optimization controller, which contains two inputs and one output, and the function implemented is to extract and compare and analyze the semantic information of its input, and continuously optimize the network structure and performance parameters of the generation module and the content module; the content discriminator A and the content discriminator B have the same function and structure, both contain one input and one output, and their role is to extract content features from their inputs respectively, and obtain the semantic information feature data of the input tensor; one input of the content discriminator A is the input 1 and its output is output content_1, the content discriminator A is the input 1 and its output is output content_1, the content discriminator B is the input 1 and the output is output content_2, the content discriminator A is the input 1 and the output is output content_3, the content discriminator B is the output content_4, the content discriminator A is the input 1 and the output is output content_5, the content discriminator B is the output content_6, the content discriminator B is the output content_7, the content discriminator A is the input 1 and the output is output content_8, the content discriminator B is the output content_9, the content discriminator B is the output content_1, the content discriminator A is the input 1 and the output is output content_1, the content discriminator B is the output content_1, the content discriminator A is the input 1 and the output is output content_2 ... B is the output content_2, the content discriminator A is the input One input of content discriminator B is the input G2 and its output is output content_2; the content discriminator A and content discriminator B are both composed of convolutional layer A9, convolutional layer A10, convolutional layer A11, and convolutional layer A12 connected in series in sequence, and the function of the convolutional layer is to extract deep-level image feature information of its input tensor; the content module optimization controller contains two inputs and three implicit outputs, the two inputs are the output content_1 and the output content_2, the three implicit outputs are output content feedback 1, output content feedback 2, and output content feedback 3, and the output content feedback 3 is the output of the content module; the content module optimization controller and the texture module optimization controller have the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is
[0040]
[0041] Where D is the content discriminator A, content discriminator B, G2, I p are the input G2 and input 1, i and j are the positions of the image data, H, W, C are the length, width, and number of channels of the input of D, λ4∈[1.5*10 3 , 2.5*10 3 ] is the weight coefficient of the content module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the content discriminator A, content discriminator B and generation module respectively, so that the gradient of its neurons converges as a whole. The principle of optimization is to make the loss function L content The value of keeps decreasing;
[0042] The structure module is composed of a segmentation layer, a coloring layer, a structure discriminator A, a structure discriminator B and a structure module optimization controller, which contains an input and an implicit output, the one input is input G2, which corresponds to the output G2, and the function of this module is to extract and compare the visual information of the object structure of its input, and to back-propagate the comparison results to continuously optimize the network structure and performance parameters of it and the generation module; the segmentation layer, the coloring layer, and the structure discriminator A all contain an input and an output and are connected in series in sequence, the input of the segmentation layer is the input G2 of the structure module, and the output of the structure discriminator A is the output structure_1; the structure discriminator B contains an input and an output, the input is G2, and the output is structure_2; the structure discriminator A, the structure discriminator B and the Content discriminator A and content discriminator B have exactly the same structure, and their function is to extract the visual information of the input tensor and obtain the image data of the sparse color blocks and clear edges of the input tensor; the function of the segmentation layer is to divide the input G2 into 200 small areas, and the function of the coloring layer is to color the 200 small areas; the structure module optimization controller contains two inputs and three implicit outputs, the two inputs are the output structure_1 and the output structure_2, the three implicit outputs are output structure feedback 1, output structure feedback 2 and output structure feedback 3, and the output structure feedback 3 is the output of the structure module; the structure module optimization controller and the texture module optimization controller have exactly the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is
[0043]
[0044] Among them, D is the structure discriminator A, the structure discriminator B, f f is the segmentation layer, f z is the coloring layer, i, j are the image data positions, H, W, C are the height, width, and number of channels of the input G2, λ5∈[0,10 5 ] is the weight coefficient of the structure module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the structure discriminator A, structure discriminator B and generation module respectively, so that the gradient of the neurons converges as a whole. The principle of optimization is to make the loss function L structure The value of is decreasing and converging;
[0045] The structure discriminator A, structure discriminator B, content discriminator A, content discriminator B have exactly the same structure and function, and are all composed of convolution layer A9, convolution layer A10, convolution layer A11, and convolution layer A12 connected in series in sequence, and are used to extract the features of their input tensors and obtain the deep information of the image; the convolution layer A9 is composed of convolution layer O, leaky relu activation function layer, convolution layer O, leaky relu activation function layer, and maximum pooling layer connected in series in sequence; the convolution layer A10 is composed of convolution layer P, leaky relu activation function layer, convolution layer P, leaky relu activation function layer, and maximum pooling layer connected in series in sequence; the convolution layer A11 is composed of convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer and maximum pooling layer are connected in series in sequence; the convolution layer A12 is composed of convolution layer R, leaky relu activation function layer, convolution layer R, leaky relu activation function layer, convolution layer R, leaky relu activation function layer, convolution layer R, and maximum pooling layer connected in series in sequence; the functions of the convolution layer O, convolution layer P, convolution layer Q, and convolution layer R are to extract features of their input tensors; the function of the leaky relu activation function layer is to reduce the data of the input tensor less than 0 to 0.2 times of the original; the function of the maximum pooling layer is to reduce the size of the input tensor to half of the original;
[0046] Furthermore, the generative adversarial neural network based on image creation is characterized in that the network is built based on the Tensorflow platform, the optimizer adopts the AdamOptimizer algorithm, and the algorithm contains a learning rate parameter, a first momentum gradient descent factor and a small constant; the summation operation adopts the add algorithm;
[0047] The convolution layer A, convolution layer B, convolution layer C, convolution layer D, convolution layer E, convolution layer F, convolution layer G, convolution layer H, convolution layer I, convolution layer J, convolution layer K, convolution layer L, convolution layer M, convolution layer N, convolution layer O, convolution layer P, convolution layer Q, convolution layer R, convolution layer S, convolution layer T, convolution layer U, convolution layer V, and convolution layer W have the same structure and function, and the construction parameters of their structure include input tensor, convolution kernel size, number of convolution kernels, step size of convolution kernel, filling method of convolution kernel, bias term of convolution kernel, and initial value of convolution kernel, and their function is to extract features of their input tensor; the convolution layers have different input tensors, convolution kernel sizes, and convolution kernels. The number, step size of the convolution kernel, and filling method of the convolution kernel are "SAME", "VALID", and "REFLECT"; the initial value of the convolution kernel is a standard normal distribution with a standard deviation of 0.01, and the initial value is generated by the Initializers.Xavier_initializer algorithm; the initial value of the bias term of the convolution kernel is 0, and the initial value is generated by the Init_ops.Zeros_initializer algorithm; the convolution layer adopts the nn.conv2d algorithm, and its output is the sum of the convolution operation and the bias term, and the sum adopts the add algorithm; The construction parameters of the convolution layer A are: the size of the convolution kernel is 7*7, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer B are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer C are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer D are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1 , 2, 2, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer E are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer F are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer G are that the size of the convolution kernel is 7*7, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 3;The construction parameters of the convolution layer H are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer I are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer J are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer K are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], and the number of convolution kernels is The filling method is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer L are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer M are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer N are that the size of the convolution kernel is 1*1, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 1; the construction parameters of the convolution layer O are that the size of the convolution kernel is 3* 3. The step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer P are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer Q are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 256; the construction parameters of the convolution layer R are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 5 12; the construction parameters of the convolution layer S are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the top, bottom, left and right, and the number of convolution kernels is 16; the construction parameters of the convolution layer T are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right, and the number of convolution kernels is 32; the construction parameters of the convolution layer U are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right, and the number of convolution kernels is 64;The construction parameters of the convolution layer V are as follows: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right corner, and the number of convolution kernels is 128; the construction parameters of the convolution layer W are as follows: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the top, bottom, left and right corners, and the number of convolution kernels is 1;
[0048] The convolutional layers 1, 2, 3 and 4 are convolutional kernels with specific requirements, whose convolutional kernel size is 3*3, whose step size is [1, 1, 1, 1], and whose filling mode is "VAILD"; the convolutional kernels of the convolutional layer 1 are [[[0, -1, 0], [0, 0, 0], [0, 1, 0]], [[0, -2, 0], [0, 0, 0], [0, 2, 0]], [[0, -1, 0], [0, 0, 0], [0, 1, 0]]], and the convolutional kernels of the convolutional layer 2 are [[[0, -1, 0], [0, 0, 0], [0, 1, 0]]], [[0, -2, 0], [0, 0, 0], [0, 2, 0]], [[0, -1, 0], [0, 0, 0], [0, 1, 0]]], [[0, The convolution kernels are [[[0, 0, 0], [-1, 0, 1], [0, 0, 0]], [[0, 0, 0], [-2, 0, 2], [0, 0, 0]], [[0, 0, 0], [-1, 0, 1, [0, 0, 0]]], and the convolution kernels of convolution layer 3 are [[[0, -1, 0], [0, 2, 0], [0, -1, 0]], [[0, -2, 0], [0, 4, 0], [0, -2, 0]], [[0, -1, 0], [0, 2, 0], [0, 1, 0]]], The convolution kernel of convolution layer 4 is [[[0, 0, 0], [-1, 2, -1], [0, 0, 0]], [[0, 0, 0], [-2, 4, -2], [0, 0, 0]], [[0, 0, 0], [-1, 2, -1], [0, 0, 0]]]; the smoothing filter layer adopts the nn.depthwise_conv2d algorithm, and its construction parameters include the input tensor, the size of the convolution kernel, the step size of the convolution kernel, and the filling method of the convolution kernel. The size of the convolution kernel of the smoothing filter layer is 3 *3. The step size of the convolution kernel is [1, 1, 1, 1], and the filling mode of the convolution kernel is "SAME"; the initial value of the convolution kernel of the smoothing filter layer is a standard normal distribution with a standard deviation of 0.01, and the initial value is generated by the Initializers.Xavier_initializer algorithm; the initial value of the bias item of the convolution kernel of the smoothing filter layer is 0, and the initial value is generated by the Init_ops.Zeros_initializer algorithm;
[0049] The color decomposition layer is based on the relation
[0050]
[0051] To achieve its function, R I、 G I、 B I R2, G2, and B3 are two inputs, which are called the three-channel data of input image I1 and input image I2. I、 G I , B I is the data of the red, green and blue channels of the image I1, R2, G2 and B2 are the data of the red, green and blue channels of the image I2 respectively, R w , G w , B w are weight coefficients of red, green and blue channels respectively; the segmentation layer adopts Python.Skimage.Felzenszwalb segmentation algorithm, which divides the image into some small areas, each area has similar pixels and positions; the coloring layer colors each image area segmented by the segmentation layer, and if the standard deviation of the pixel values in the area is less than 20, the average color is used for coloring; if the standard deviation is between 20 and 40, the mixture of the average and median colors is used for coloring; if the standard deviation is greater than 40, the median color is used for coloring;
[0052] The maximum pooling layer adopts the nn.max_pool algorithm, and the construction parameters of the maximum pooling layer include the input tensor, the size of the convolution kernel, the step size of the convolution kernel, and the filling method of the convolution kernel. The parameters of all the maximum pooling layers are that the size of the convolution kernel is 2*2, the step size of the convolution kernel is [1, 2, 2, 1], and the filling method of the convolution kernel is "SAME"; the normalization layer is implemented using the layer_norm algorithm; the compression layer adopts the clip_by_value algorithm, and its construction parameters include the input tensor and the compression range; the linear layer 1 adopts the shape and Image.Resize_bilinear algorithms, which expand the length and width of the input tensor by 2 times.
[0053] The present invention also proposes a method for generating cartoon face images based on image creation using a generative adversarial neural network, which is characterized by comprising the following steps:
[0054] Step 1: Construct a real face color image dataset D1 and a cartoon face color image dataset D2 of size M*N; the images in the dataset D1 are real images collected by the device, and the images in the dataset D2 are cartoon images created by artists, which have clear contours, sparse color blocks, more details and other features, and soft strokes, more thick lines, smooth object surfaces, smooth colors in non-edge areas, and a soft overall image effect;
[0055] Step 2: Select any pair of images from the data set D1 and the data set D2 in step 1, respectively referred to as input 1 and input 2, and then input the input 1 into the generation module described in claim 1 to obtain the output G1 of the generation module;
[0056] Step 3: Input the output G1 obtained in step 2 and the input 1 into the enhancement module described in claims 1-2 to obtain the output G2 of the enhancement module;
[0057] Step 4: Input the output G2 obtained in step 3 and the input 2 described in step 1 into the texture module of the cartoon style extraction module described in claims 1-2, and the texture module optimization controller of the texture module optimizes the texture module and the generation module according to its implicit output; input the output G2 obtained in step 3 and the input 2 described in step 1 into the surface module of the cartoon style extraction module described in claims 1-2, and the surface module optimization controller of the surface module optimizes the surface module and the generation module according to its implicit output; input the output G2 obtained in step 3 and the input 2 described in step 1 into the cartoon style extraction module described in claims 1-2 In the local texture module of the block, the local texture module optimization controller of the local texture module optimizes the table local texture module and the generation module according to its implicit output; G2 in step 3 and the input 1 described in step 1 are input into the content module of the real face extraction module described in claims 1-2, and the content module optimization controller of the content module optimizes the content module and the generation module according to its implicit output; the output G2 obtained in step 3 is input into the structure module of the real face extraction module described in claims 1-2, and the structure module optimization controller of the structure module optimizes the structure module and the generation module according to its implicit output;
[0058] Step 5: Repeat steps 2-4 to train the network described in claims 1-2 for multiple times, and save the training results every KK∈[30, 1000] training times; after the total number of training times reaches iteration∈[50000, 200000] times, the final face cartoon style transfer model M is obtained;
[0059] Step 6: The real face image to be processed is taken as input 1 and input into the model M obtained in step 5. The output G2 obtained by processing the generation module and enhancement module model in the model M is the high-quality cartoon-style face image quickly generated by the model M.
[0060] Furthermore, the method for extracting cartoon style of human face using generative adversarial neural network based on image creation is characterized in that the method is implemented using Python language and Tensorflow platform; the learning rate parameter in claim 2 is 0.0002, the first momentum gradient descent factor is 0.5, and the small constant is 1*10 -8 The color decomposition layer described in claims 1-2 uses the Split module to separate the three channels of the input image I1 and the input image I2, and uses the Random.Uniform module to generate the random weight parameter R w , G w , B w ; The dataset D1 described in step 1 is 5000 images in the FFHQ dataset, and the dataset D2 is cartoon face images captured from movies produced by Kyoto Animation; the images in the datasets D1 and D2 described in step 1 are color images in JPG format with sizes M=256 and N=256; KK=500, iteration=100000 in step 5; the texture module, surface module, local texture module, content module and structure module described in step 4 involve the λ described in claim 1 1、 λ2, λ3, λ4, λ5, preferably, λ1=10, λ2=1, λ3=300, λ4=2*10 3 ,λ5=2*10 3 ; The cartoon-style face image obtained in step 6 is a high-quality cartoon face image of a specific cartoon style having the content features of input 1 described in step 1. The image has clear contours, more details, soft strokes, more thick lines, a smooth surface, smooth colors in non-edge areas, and the entire image is soft.
[0061] The invention provides a generative adversarial neural network based on image creation and a cartoon face image generation method, which also has the following characteristics:
[0062] 1. The present invention provides a generative adversarial neural network based on image creation and a cartoon face creation method, which are built based on Python language and Tensorflow deep learning framework.
[0063] 2. The face cartoon style transfer model generated by this method only needs to output a real face image when used, and the format can be JPG or PNG.
[0064] 3. The method provided by the present invention is constructed for the image creation process, and the image creation is divided into five parts: texture, surface, local texture, content and structure, and the generated cartoon image is optimized according to these five parts;
[0065] 4. The generation module provided by the present invention comprises 18 convolutional layers, 12 activation function layers, 6 operation layers, 2 linear layers and one compression layer; the texture discriminator A, texture discriminator B, surface discriminator A and surface discriminator B of the texture module and the surface module provided by the present method each comprise 7 convolutional layers and 6 activation layers; the local texture discriminator A and the local texture discriminator B of the local texture module provided by the present method each comprise 5 convolutional layers, 4 activation function layers and 4 normalization layers; the content discriminator A, content discriminator B, structure discriminator A and structure discriminator B of the content module and the structure module provided by the present method each comprise 12 convolutional layers, 12 activation function layers and 4 maximum pooling layers; the generative adversarial neural network is constructed by controlling the connection order of different modules, and the library used is Tensorflow.
[0066] 5. The operation of the color decomposition layer of the texture module and the local texture module of the present invention is as follows: Let the input images be images I1 and I2, and use the tf.split module to separate the three channels of images I1 and I2 to obtain data R I , G I , B I and R2, G2, B3, using the tf.random.uniform module to construct a randomly generated number R w , G w , B w , and then use the relation
[0067]
[0068]
[0069] Get the output data of this layer, where R I , G I , B I is the data of the red, green, and blue channels of image I1, R2, G2, and B2 are the data of the red, green, and blue channels of image I2, and R w、 G w , B w are the weights of the red, green, and blue channel data of the image.
[0070] 6. The segmentation layer of the structural module of the present invention adopts the Felzenszwalb segmentation algorithm, which is implemented by the Python skimage module. The Felzenszwalb segmentation algorithm divides the image into a number of small areas, each of which has similar pixels and positions;
[0071] 7. The coloring layer of the structural module of the present invention colors each image region segmented by the segmentation layer. If the standard deviation of the pixel values in the region is less than 20, the average color is used for coloring; if the standard deviation is between 20 and 40, a mixture of the average and median colors is used for coloring; if the standard deviation is greater than 40, the median color is used for coloring.
[0072] 8. The optimization controller of each module provided by the present invention contains a loss function and an optimizer, which reduces the value of the loss function through multiple trainings to make the output image more in line with our requirements. The total loss function of the generative adversarial neural network is composed of a texture loss function, a surface loss function, a local texture loss function, a structural loss function, and a content loss function.
[0073] 9. The method of the present invention is more robust and efficient, and can generate desired cartoon face images according to people's needs, making up for the shortage of cartoon faces in the existing market. It can be applied to the game field, new media field, illustration field and film and television field. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 This is a structural relationship diagram of the generative adversarial neural network based on image creation described in the present invention.
[0075] Figure 2 This is a structural relationship diagram of the generation module described in the present invention.
[0076] Figure 3 This is a structural relationship diagram of the enhancement module described in the present invention.
[0077] Figure 4 This is a structural relationship diagram of the cartoon style extraction module described in the present invention.
[0078] Figure 5 This is a structural relationship diagram of the texture module described in the present invention.
[0079] Figure 6 This is a structural relationship diagram of the surface module described in the present invention.
[0080] Figure 7 This is a structural relationship diagram of the smoothed image processing A and the smoothed image processing B described in the present invention.
[0081] Figure 8 This is a structural relationship diagram of the local texture module described in the present invention.
[0082] Fig. 9 This is a structural relationship diagram of patch extraction A and patch extraction B described in the present invention.
[0083] Fig.10This is a structural relationship diagram of the real face extraction module described in the present invention.
[0084] Fig.11 This is a structural relationship diagram of the content module described in the present invention.
[0085] Fig.12 This is a structural relationship diagram of the structural modules described in the present invention.
[0086] Fig.13 This is a training control flow chart of the generative adversarial neural network based on image creation and the method for generating cartoon faces described in the present invention.
[0087] Fig.14 A natural face image (left) and a cartoon face image (right) inputted according to the first embodiment of the present invention.
[0088] Fig.15 A natural face image (left) and a corresponding cartoon face image (right) as input to the generative adversarial neural network model of Example 1 of the present invention. DETAILED DESCRIPTION
[0089] The following is a further description of the specific implementation of the present invention in conjunction with the accompanying drawings and examples. The following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0090] Figure 1-12 The main structure and structural relationship flow chart of a generative adversarial neural network based on image creation of the present invention are constituted; Fig.13 The invention discloses a training control flow chart of a generative adversarial neural network and a method for generating cartoon faces based on image creation, including the training process of the network and the construction and use of the network model. Combined with the technical solutions of the generative adversarial neural network and the method for generating cartoon face images described in the invention, a specific structure for constructing a neural network for relevant application fields can be constructed, and a network model capable of solving specific problems can be obtained by training the constructed neural network with corresponding data sets, so as to solve a series of cartoon style missing problems such as missing semantic content, unclear cartoon texture, incomplete structure, etc. existing in cartoon face images generated by existing image style transfer technologies, thereby alleviating the workload of artists and quickly creating high-quality cartoon face images to meet the aesthetic needs of different groups of people.
[0091] The following embodiments provided by the present invention are based on Figure 1-13 The structure, control flow chart and technical solution provided by the present invention are implemented, and the specific embodiments are as follows:
[0092] Embodiment 1
[0093] First, an image training set D1 is constructed, which contains 5000 natural face images, and an image training set D2 is constructed, which contains 5000 cartoon face images. The image format of both is JPG, the image size is M*N pixels, M=256, N=256, and the pixel bit depth is 24.
[0094] Then, a generative adversarial neural network based on the image creation process provided by the present invention is used, see Figure 1-13 , and the technical solution provided by the present invention, to build the network of this embodiment; the generative adversarial neural network of this embodiment is implemented using Python language and Tensorflow platform; the network structure and parameters of this embodiment are the same as those of the technical solution of the present invention. The color decomposition layer of the network structure of this embodiment uses the tf.split module to separate the three channels of its input, and uses the tf.random.uniform module to generate the weight parameter R w , G w , B w ; The weight parameters of each loss function in this embodiment are λ1=10, λ2=1, λ3=300, λ4=2*10 3 ,λ5=2*10 3 The optimizer of the optimization controller adopts the AdamOptimizer algorithm, and the learning rate parameter of the algorithm is 0.0002, the first momentum gradient descent factor is 0.5, and the small constant is 1*10 -8 .
[0095] Then, the generative adversarial neural network and cartoon face image generation method based on image creation provided by the present invention are used to construct the face cartoon style transfer model of the embodiment. The training control flow chart is shown in Fig.13 , a natural face image a (see Fig.14 The image on the left is input into the generation module, which processes the input image and outputs it.
[0096] Then generate the output of the module and Fig.14 The left picture is input to the enhancement module of the present invention to obtain the output of the enhancement module.
[0097] Then the output of the enhancement module and a cartoon face image b in the image training set D2 (see Fig.14 The right figure) is input into the texture module, surface module, and local texture module of the cartoon style extraction module to realize the Fig.14 Training optimization of texture module, surface module, local texture module and generation module.
[0098] Then, the output of the enhancement module and the natural face image a in the image training set D1 are input into the content module and the structure module of the real face extraction module to realize Fig.14 The training and optimization of the content module, structure module and generation module of this embodiment.
[0099] According to the technical solution proposed by the present invention and Fig.13 The network model training control flow chart performs multiple iterative training on all images in the data set D1 and the data set D2, KK=500, iteration number=100000 times, and finally obtains the neural network model M of this example.
[0100] Finally, the real face image c (see Fig.15 Left), which is input into the obtained image-based generative adversarial neural network model, and the output of the model is Fig.15 The cartoon face image of the real face image c shown in the right figure. The obtained cartoon face image is a high-quality cartoon face image, which has a clear outline, more details, soft strokes, more thick lines, a smooth surface, smooth colors in non-edge areas, and a soft image.
[0101] Embodiment 2
[0102] First, an image training set D1 and an image training set D2 are constructed. The image formats of the two sets are PNG, the image size is M*N pixels, M=256, N=256, and the pixel bit depth is 24.
[0103] Then, a generative adversarial neural network based on the image creation process provided by the present invention is used, see Figure 1-13 , and the technical solution provided by the present invention, to build the network of this embodiment; the generative adversarial neural network of this embodiment is implemented using Python language and Tensorflow platform; the network structure and parameters of this embodiment are the same as those of the technical solution of the present invention. The color decomposition layer of the network structure of this embodiment uses the tf.split module to separate the three channels of its input, and uses the tf.random.uniform module to generate the weight parameter R w , G w , B w ; The weight parameters of each loss function in this embodiment are λ1∈[0,10], λ2∈[1,2], λ3∈[200,400], λ4∈[1.5*10 3 , 2.5*10 3 ],λ5∈[1.5*10 3 , 2.5*10 3], the optimizer of the optimization controller adopts the AdamOptimizer algorithm.
[0104] Then, the generative adversarial neural network and face cartoon style extraction method based on the image creation process provided by the present invention are used to construct the face cartoon style transfer model of the embodiment. The training control flow chart is shown in Fig.13 , a natural face image a in the image data training set D1 is input into the generation module, and the generation module processes the input image and outputs it.
[0105] Then, the output of the generation module and the natural face image a are input into the enhancement module of the present invention to obtain the output of the enhancement module.
[0106] The output of the enhancement module and a cartoon face image in the image training set D2 are then input into the texture module, surface module, and local texture module of the cartoon style extraction module to achieve training optimization of the texture module, surface module, local texture module, and generation module based on the natural face image a and the cartoon face image b of this embodiment.
[0107] The output of the enhancement module and the natural face image a in the image training set D1 are then input into the content module and structure module of the real face extraction module to achieve training optimization of the content module, structure module and generation module of this embodiment based on the natural face image a and the cartoon face image b.
[0108] According to the technical solution proposed by the present invention and Fig.13 The network model training control flow chart performs multiple iterative training on all images in the data set D1 and the data set D2, and saves the training results every KK∈[30, 1000] training times; after the total number of training times reaches iteration∈[50000, 200000] times, the final face cartoon style transfer model M is obtained.
[0109] Finally, the real face image c that needs to be style-converted is input into the obtained generative adversarial neural network model based on image creation, and the output of the model is a cartoon face image of the real face image c. The obtained cartoon face image is a high-quality cartoon face image with clear contours, more details, soft strokes, more thick lines, smooth surface, smooth color in non-edge areas, and a soft image.
[0110] The above embodiments only illustrate and explain the implementation methods of the present invention. Any modifications and variations made according to the scope of the claims of the present invention belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A generative adversarial neural network based on image creation, characterized in that: The network is constructed based on the image creation process and is used to quickly generate high-quality cartoon face images; the network contains two inputs, namely input 1 and input 2, the former of which is the input natural face image and the latter is the input cartoon style face image; the network is composed of a generation module, an enhancement module, a cartoon style extraction module, and a real face extraction module; the generation module contains one input and one output, the one input is the input 1, and the one output is output G1, which is an image G1 generated according to the input 1 with desired cartoon style features including texture, surface, local texture features, etc.; the enhancement module contains two inputs and one output, the two inputs are the input 1 and the output G1, and the one output is output G2, which combines the two inputs of the enhancement module. The cartoon style features of the output G1 are assigned to the input 1, and the cartoon style image data with clearer image contours, more details, softer brush strokes, more thick lines and smoother surface is obtained; the cartoon style extraction module contains two inputs and three implicit outputs, the two inputs are the output G2 and the input 2, the module extracts and compares and analyzes the texture, surface and local texture features of the two inputs, and continuously optimizes and regulates the network structure and performance parameters of the generation module through implicit outputs; the real face extraction module contains two inputs and two implicit outputs, the two inputs are the output G2 and the input 1, the module extracts and compares and analyzes the content and structural features of the two inputs, and continuously optimizes and regulates the structure and performance parameters of the generation module through implicit outputs; The generation module is composed of convolution layer A, leaky relu activation function layer, convolution layer A1, convolution layer A2, convolution layer A3, operation layer 1, convolution layer A4, operation layer 2, convolution layer A5, operation layer 3, convolution layer A6, operation layer 4, bilinear layer 1, operation layer 5, convolution layer A7, bilinear layer 2, operation layer 6, convolution layer A8, and compression layer connected in series in sequence; the operation layer 1, operation layer 2, operation layer 3, operation layer 4, operation layer 5, and operation layer 6 each have two inputs and one output, and the function realized is to perform a sum operation on the two inputs; the two inputs of the operation layer 1 are the convolution layer A2, The output of convolution layer A3, the two inputs of the operation layer 2 are the output of the operation layer 1 and the output of the convolution layer A4, the two inputs of the operation layer 3 are the output of the operation layer 2 and the output of the convolution layer A5, the two inputs of the operation layer 4 are the output of the operation layer 3 and the output of the convolution layer A6, the two inputs of the operation layer 5 are the output of the bilinear layer 1 and the output of the convolution layer A1, the two inputs of the operation layer 6 are the output of the bilinear layer 2 and the output of the activation function layer; the convolution layer A1 is composed of convolution layer B, Leaky relu activation function layer, convolution layer C, and leaky relu activation function layer are sequentially connected in series; the convolution layer A2 is composed of convolution layer D, leaky relu activation function layer, convolution layer E, and leaky relu activation function layer in series; the convolution layer A3, convolution layer A4, convolution layer A5, and convolution layer A6 are all composed of convolution layer F, leaky relu activation function layer, and convolution layer F in series; the convolution layer A7 is composed of convolution layer C and leaky relu activation function layer in series; the convolution layer A8 is composed of convolution layer F, leaky The relu activation function layer and the convolution layer G are connected in series in sequence; the functions of the convolution layers A1, A2, A3, A4, A5, A6, A7 and A8 are to extract features from their inputs and obtain deep feature information of the image; the functions of the convolution layers A, B, C, D, E, F and G are to extract features from their inputs; the function of the leaky relu activation function layer is to adjust the data whose input tensor is less than zero to 0.2 times of the original; the bilinear layer 1 is composed of the convolution layers C, Leaky The relu activation function layer and the linear layer 1 are connected in series in sequence, and the linear layer 1 is used to adjust the size of its input; the bilinear layer 2 is composed of the convolution layer F, the leakyrelu activation function layer, and the linear layer 1 in series in sequence; the bilinear layer 1 and the bilinear layer 2 have the same function, both of which are to extract features and change the size of their input tensors; the compression layer is used to adjust the input tensor data to [-1, 1]; The enhancement module contains two inputs and one output, the two inputs are the input 1 and the input G1, the input G1 corresponds to the output G1 of the generation module, and the one output is the output G2; the enhancement module is composed of a computing layer 7, a computing layer 8, a computing layer 9, a computing layer 10, a computing layer 11, a computing layer 12, a computing layer 13, a computing layer 14, a smoothing filter layer and a standardization layer; the function of the computing layer 7 is to perform a square operation on the input tensor; the function of the computing layer 8 and the computing layer 9 is to perform a multiplication operation on the two input tensors; the function of the computing layer 10, the computing layer 13, and the computing layer 14 is to perform a compound operation on the input, and the compound operation is The corresponding position data of multiple input tensors are added, subtracted, multiplied and divided; the operation layer 11 and the operation layer 12 are used to perform subtraction operations on the two inputs; the smoothing filter layer has one input and one output, and its function is to perform a smoothing operation on its input, and the smoothing operation is to take the average value of the small area data in the input tensor as the central value; the standardization layer has one input and one output, and its function is to reduce the data at all positions of the input tensor, and the reduction operation is to divide the data at all positions of the input tensor by the size of the input tensor; the function of the enhancement module is to use G1 as the guiding information to perform cartoon style enhancement on input 1, which is achieved through the following process: Process 1: The input 1 is processed in series by the operation layer 7, the smoothing filter layer, and the normalization layer to obtain the output tensor data Q1; Process 2: The input 1 is processed in series with the smoothing filter layer and the normalization layer to obtain the output tensor data Q2; the input G1 is processed in series with the smoothing filter layer and the normalization layer to obtain the output tensor data Q3; Process 3: input the obtained data Q2 and data Q3 into the operation layer 9, and obtain tensor data Q4=Q2*Q3 through multiplication operation of the operation layer 9; Process 4: The input 1 and input G1 are input to the operation layer 8, and then sequentially processed in series by the smoothing filter layer and the normalization layer to obtain the output tensor data Q5; Process 5: Input the obtained data Q1 and Q4 to the operation layer 11, and obtain tensor data Q6=Q1-Q4 through subtraction operation of the operation layer 11; input the obtained data Q4 and Q5 to the operation layer 12, and obtain tensor data Q7=Q5-Q4 through subtraction operation of the operation layer 12; Process 6: Input the obtained data Q6 and data Q7 to the operation layer 10, and obtain tensor data QB=Q7 / (Q6+0.01)=Q8 through compound operation of the operation layer 10; Process 7: Input the obtained data Q8 to the smoothing filter layer and the standardization layer, and obtain the output tensor data Q9 through the series processing; Process 8: Input the obtained data Q2, data Q3 and data Q8 to the operation layer 13, and obtain tensor data Q10 = Q3-Q2*Q8 through compound operation of the operation layer 13; Process 9: Input the obtained data Q10 into the smoothing filter layer and the standardization layer, and obtain the tensor data Q11 through the series processing; Process 10: Input the input 1 and the obtained data Q9 and data Q11 to the operation layer 14, and obtain the output tensor data G2 = the input 1*Q9+Q11 through compound operation processing, that is, the output G2; The cartoon style extraction module is composed of a texture module, a surface module and a local texture module. Each module contains two inputs and one implicit output. The two inputs are the input 2 and the input G2. The input G2 corresponds to the output G2 of the enhancement module. The module extracts and compares and analyzes the texture, surface and local texture features of its input, and then feeds the results back to the generation module and adjusts the corresponding submodules of the module, so as to continuously optimize and regulate its network structure and performance parameters. The texture module is composed of a color decomposition layer, a texture discriminator A, a texture discriminator B and a texture module optimization controller, which contains two inputs and one implicit output. The function of the module is to extract and compare the details and structural edge features of the grayscale images of its two inputs, back-propagate the comparison results, and continuously optimize the structure and performance parameters of the corresponding submodules of the module and the generation module; the color decomposition layer contains two inputs and two outputs, the two inputs are the input G2 and the input 2, and the two outputs are output gray_fake and output gray_cartoon, which correspond to the grayscale image of the input G2 and The grayscale image of the input 2, the function of the color decomposition layer is to convert the two color image data of its input into grayscale image data; the texture discriminator A and the texture discriminator B have the same structure and function, and both contain an input and an output; the input of the texture discriminator A is the output gray_fake, and its output is output texture_1, the input of the texture discriminator B is the output gray_cartoon, and its output is output texture_2; the functions of the texture discriminator A and the texture discriminator B are to extract the texture feature information such as the details and structural edges of their inputs, which are both composed of the convolutional layer H, the Leaky relu activation function layer, convolution layer I, leaky relu activation function layer, convolution layer J, leaky relu activation function layer, convolution layer K, leaky relu activation function layer, convolution layer L, leaky relu activation function layer, convolution layer M, leaky relu activation function layer, and convolution layer N are sequentially connected in series; the functions of the convolution layer H, convolution layer I, convolution layer J, convolution layer K, convolution layer L, convolution layer M, and convolution layer N are all to extract the details and structural edge features of their inputs; the texture module optimization controller contains two inputs and three implicit outputs, the two inputs are the output texture_1 and the output texture_2, the three implicit outputs are output texture feedback 1, output texture feedback 2 and output texture feedback 3, and the output texture feedback 3 is the implicit output of the texture module; the texture module optimization controller is composed of a loss function and an optimizer, the loss function contains two inputs and one output, and the one output is the input of the optimizer; the loss function of the texture module optimization controller is l texture =λ1[logD(f rcs (I C ))+log(1-D(f rcs (G2)))] Among them, f rcs is the color decomposition layer, I C , G2 are the input 2 and input G2 respectively, D is the texture discriminator A or texture discriminator B, log is the logarithmic operation, l texture is the texture loss value, λ1∈[0,10] is the weight coefficient of the texture module; the optimizer optimizes and adjusts the network structure and performance parameters of the texture discriminator A, texture discriminator B and generation module through its three implicit outputs to make the gradient of the neurons converge as a whole. The principle of optimization and adjustment is to make the loss function l texture Continuously descending and converging; The surface module is composed of smooth image processing A, smooth image processing B, surface discriminator A, surface discriminator B and surface module optimization controller, which contains two inputs and one implicit output, and its function is to extract and compare the surface features of the pixel smooth area of the input G2 and input 2, and then back-propagate the comparison results and feed them back to the generation module and the corresponding submodules of the module to continuously optimize and regulate its network structure and performance parameters; the smooth image processing A and the smooth image processing B have the same structure and function, both contain one input and one output, and their function is to smooth and denoise their inputs to obtain smoother and clearer image data; the input of the smooth image processing A is the input G2, which corresponds to the output G2, and its output is output blur_fake, which means the blurred image of the input G2; the input of the smooth image processing B is the input 2, and its output is output blur_cartoon, Its meaning is the blurred image of the input 2; the surface discriminator A, the surface discriminator B and the texture discriminator A have exactly the same structure, and all contain an input and an output, which are surface feature image data for extracting the input while removing edges, details, and textures and retaining colors; the input of the surface discriminator A is the output blur_fake and its output is the output surface_1, the input of the surface discriminator B is the output blur_cartoon and its output is the output surface_2; the surface module optimization controller contains two inputs and three implicit outputs, the two inputs are the output blur_fake and the output blur_cartoon, the three implicit outputs are surface feedback 1, surface feedback 2 and surface feedback 3, and the surface feedback 3 is the implicit output of the surface module; the surface module optimization controller is composed of a loss function and an optimizer connected in series in sequence, and the loss function is l surface =λ2[logD(f dgf (I C ))+log(1-D(f dgf (G2))] Among them, f dgf are the smooth image processing A and the smooth image processing B, D are the surface discriminator A and the surface discriminator B, log is the logarithmic operation, l surface is the surface loss value, λ2∈[1,2] is the weight coefficient of the surface module; the optimizer optimizes the network structure and performance parameters of the surface discriminator A, surface discriminator B and generation module through its three implicit outputs, so that the gradient of the neurons converges as a whole. The principle of optimization is to make the loss function l surface The value of is decreasing and converging; The smooth image processing A and the smooth image processing B are both composed of a smooth filter layer, a standardization layer, a computing layer 15, a computing layer 16, a computing layer 17, a computing layer 18, a computing layer 19, and a computing layer 20; the computing layer 15 and the computing layer 16 each have one input and one output; the computing layer 17 has two inputs and one output, and implements a subtraction operation on its input; the computing layer 18 has one input and one output, the computing layer 19 has two inputs and one output, and the computing layer 20 has three inputs and one output, and the function implemented is a composite operation on its input; the smooth filter layer has one input and one output, and is used to smooth its input, and the smoothing process refers to taking the average value of the data of the small area in the input tensor as the central value; the standardization layer has one input and one output, and its function is to reduce the data at all positions of the input tensor, and the reduction operation is to divide the data at all positions of the input tensor by the size of the input tensor; the function of the smooth image processing A and the smooth image processing B is to keep the edge of the input image clear, which is achieved through the following process: Process 1: Let the input be image data I, and process the image I in series through the smoothing filter layer, the normalization layer and the operation layer 16 to obtain output data P1; Process 2: The data image I is processed in series by the operation layer 15, the smoothing filter layer, and the normalization layer to obtain output data P2; Process 3: input the obtained data P1 and data P2 to the operation layer 17, and obtain output data P3=P2-P1 through subtraction operation of the operation layer 17; Process 4: input the obtained data P3 to the operation layer 18, and obtain the output data P4=P3 / (P3+ε1) through the compound operation of the operation layer 18, wherein ε1=0.01 is a small constant used to avoid the denominator being zero; Process 5: input the obtained data P4 and data P1 to the operation layer 19, and obtain output data P5=P1-P4*P1 through compound operation of the operation layer 19; Process 6: input the obtained data P4 to the smoothing filter layer and the standardization layer, and then sequentially connect them in series to obtain the output data P6; Process 7: input the obtained data P5 to the smoothing filter layer and the standardization layer, and then sequentially connect them in series to obtain the output data P7; Process 8: The obtained data P6, data P7 and the image data I are input to the operation layer 20, and the output data P8=P6*image I+P7 is obtained through the compound operation of the operation layer 20. The data P8 is the output of the smoothing image processing A and the smoothing image processing B. The local texture module is composed of a patch extraction layer A, a patch extraction layer B, a color decomposition layer, a local texture discriminator A, a local texture discriminator B and a local texture module optimization controller, which contains two inputs and one implicit output. The function of the module is to extract and compare the details and structural edge local texture features of the corresponding grayscale image of the small area image of its input, and to back-propagate the comparison results, and continuously optimize the structure and performance parameters of the corresponding submodules of the module and the generation module; the patch extraction layer A and the patch extraction layer B have exactly the same structure and function, both contain one input and one output, and their function is to extract the small area image data in the input G2 and the input 2 respectively; the input of the patch extraction layer A is the input G2 and its output is output G2_patches, the input of the patch extraction layer B is the input 2 and its output is cartoon_patches; the color decomposition layer has two inputs and two outputs, the two inputs are the output G2_patches and the output cartoon_patches, and the two outputs are input Output G2_patche_gray and output cartoon_patches_gray, which are the small area grayscale images of their inputs respectively; the local texture discriminator A and the local texture discriminator B have exactly the same structure and function, and both contain one input and one output; the input of the local texture discriminator A is the output G2_patche_gray and its output is output local_1, the input of the local texture discriminator B is the output cartoon_patches_gray and its output is local_2; the local texture module optimization controller contains two inputs and three implicit outputs, the two inputs are the output local_1 and the output local_2, the three implicit outputs are local texture feedback 1, local texture feedback 2 and local texture feedback 3, and the local texture feedback 3 is the implicit output of the local texture module; the local texture module optimization controller and the texture module optimization controller have exactly the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is L local =λ3[logD(f rcs (f p (I C ))+log(1-D(f rcs (f p (G2))))] Among them, f p are the patch extraction layer A and the patch extraction layer B, f rcs is the color decomposition layer, D is the local texture discriminator A, the local texture discriminator B, log is the logarithmic operation, L local is the local texture loss value, λ3∈[200,400] is the weight coefficient of the local texture module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the local texture discriminator A, the local texture discriminator B and the generation module respectively, so that the gradient of the neurons of the local texture discriminator A and the generation module converges as a whole. The principle of the optimization process is to make the loss function L local The value of keeps decreasing; The patch extraction layer A and the patch extraction layer B have the same structure and function, and are both composed of a smooth image processing C, a convolution module, an operation layer 21, a connection layer, a convolution layer 5, and an operation layer 22 connected in series in sequence; the smooth image processing C has exactly the same structure and function as the smooth image processing A and the smooth image processing B, and contains one input and one output, and the one input is the input of the patch extraction layer; the convolution module contains one input and four outputs, and its function is to extract the edge information of its input tensor to obtain the edge line image feature data of the input tensor, and the one input is the output of the smooth image processing C. The convolution module is composed of a convolution layer 1, a convolution layer 2, a convolution layer 3 and a convolution layer 4 in parallel, and the convolution layer 1, the convolution layer 2, the convolution layer 3 and the convolution layer 4 all contain one input and one input, and their inputs are all inputs of the convolution module; the operation layer 21 contains four inputs and one output, The four inputs are the four outputs of the convolution module, and the function implemented is to take the absolute value of the input and add and scale it to [0, 1] to obtain the rough edge map feature data containing the edge features of the input of the patch extraction layer A and the patch extraction layer B; the connection layer contains two inputs and one output, and the two inputs are the output of the operation layer 21 and the input of the patch extraction layer A or the patch extraction layer B, and its function is to splice the two inputs in the fourth dimension to obtain the edge map and the image data of the corresponding positions of the input of the patch extraction layer A or the patch extraction layer B overlapping each other; the convolution layer 5 contains one input and one output, and its function is to extract the small area image of the input tensor, and the function of the operation layer 22 is to calculate the pixel values of the small area image of the convolution layer 5 and sort them in descending order; the local texture discriminator A and the local texture discriminator B are both composed of the convolution layer S, the normalization layer, the Leaky relu activation function layer, convolution layer T, normalization layer, leaky relu activation function layer, convolution layer U, normalization layer, leaky relu activation function layer, convolution layer V, normalization layer, leaky relu activation function layer, and convolution layer W are sequentially connected in series, and their function is to extract and analyze the local texture features of their input, and obtain the most significant details and structural edge image feature data of the input image; the functions of the convolution layer S, convolution layer T, convolution layer U, convolution layer V, and convolution layer W are all to extract the local texture features of their input tensors, and the function of the normalization layer is to make the mean and variance of each channel of the input tensor 0 and 1 respectively; The real face extraction module is composed of a content module and a structure module; the content module contains two inputs and one implicit output, the inputs are the input 1 and input G2, and the input G2 corresponds to the output G2; The content module is composed of a content discriminator A, a content discriminator B and a content module optimization controller, which has two inputs and one output, and the function it implements is to extract and compare and analyze the semantic information of its input, and continuously optimize the network structure and performance parameters of the generation module and the content module; the content discriminator A and the content discriminator B have the same function and structure, both have an input and an output, and their role is to extract content features from their inputs respectively, and obtain the semantic information feature data of the input tensor; one input of the content discriminator A is the input 1 and its output is output content_1, one input of the content discriminator B is the input G2 and its output is output content_2; the content discriminator The discriminator A and the content discriminator B are both composed of convolutional layers A9, A10, A11 and A12 connected in series in sequence, and the function of the convolutional layer is to extract deep image feature information of its input tensor; the content module optimization controller contains two inputs and three implicit outputs, the two inputs are the output content_1 and the output content_2, the three implicit outputs are output content feedback 1, output content feedback 2 and output content feedback 3, and the output content feedback 3 is the output of the content module; the content module optimization controller and the texture module optimization controller have the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is Where D is the content discriminator A, content discriminator B, G2, I p are the input G2 and input 1, i and j are the positions of the image data, H, W, C are the length, width, and number of channels of the input of D, λ4∈[1.5*10 3 , 2.5*10 3 ] is the weight coefficient of the content module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the content discriminator A, content discriminator B and generation module respectively, so that the gradient of its neurons converges as a whole. The principle of optimization is to make the loss function L content The value of keeps decreasing; The structure module is composed of a segmentation layer, a coloring layer, a structure discriminator A, a structure discriminator B and a structure module optimization controller, which contains an input and an implicit output, the one input is input G2, which corresponds to the output G2, and the function of this module is to extract and compare the visual information of the object structure of its input, and to back-propagate the comparison results to continuously optimize the network structure and performance parameters of it and the generation module; the segmentation layer, the coloring layer, and the structure discriminator A all contain an input and an output and are connected in series in sequence, the input of the segmentation layer is the input G2 of the structure module, and the output of the structure discriminator A is the output structure_1; the structure discriminator B contains an input and an output, the input is G2, and the output is structure_2; the structure discriminator A, the structure discriminator B and the Content discriminator A and content discriminator B have exactly the same structure, and their function is to extract the visual information of the input tensor and obtain the image data of the sparse color blocks and clear edges of the input tensor; the function of the segmentation layer is to divide the input G2 into 200 small areas, and the function of the coloring layer is to color the 200 small areas; the structure module optimization controller contains two inputs and three implicit outputs, the two inputs are the output structure_1 and the output structure_2, the three implicit outputs are output structure feedback 1, output structure feedback 2 and output structure feedback 3, and the output structure feedback 3 is the output of the structure module; the structure module optimization controller and the texture module optimization controller have exactly the same structure and function, and are both composed of a loss function and an optimizer connected in series in sequence, and the loss function is Among them, D is the structure discriminator A, the structure discriminator B, f f is the segmentation layer, f z is the coloring layer, i, j are the image data positions, H, W, C are the height, width, and number of channels of the input G2, λ5∈[0,10 5 ] is the weight coefficient of the structure module; the optimizer uses its three implicit outputs to optimize the network structure and performance parameters of the structure discriminator A, structure discriminator B and generation module respectively, so that the gradient of the neurons converges as a whole. The principle of optimization is to make the loss function L structure The value of is decreasing and converging; The structure discriminator A, structure discriminator B, content discriminator A, content discriminator B have exactly the same structure and function, and are all composed of convolution layer A9, convolution layer A10, convolution layer A11, and convolution layer A12 connected in series in sequence, and are used to extract the features of their input tensors and obtain the deep information of the image; the convolution layer A9 is composed of convolution layer O, leaky relu activation function layer, convolution layer O, leaky relu activation function layer, and maximum pooling layer connected in series in sequence; the convolution layer A10 is composed of convolution layer P, leaky relu activation function layer, convolution layer P, leaky relu activation function layer, and maximum pooling layer connected in series in sequence; the convolution layer A11 is composed of convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky relu activation function layer, convolution layer Q, leaky The relu activation function layer and the maximum pooling layer are connected in series in sequence; the convolution layer A12 is composed of a convolution layer R, a leakyrelu activation function layer, a convolution layer R, a leakyrelu activation function layer, a convolution layer R, a leaky relu activation function layer, a convolution layer R, and a maximum pooling layer connected in series in sequence; the functions of the convolution layer O, the convolution layer P, the convolution layer Q, and the convolution layer R are to extract features of their input tensors; the function of the leaky relu activation function layer is to reduce the data of the input tensor less than 0 to 0.2 times of the original; the function of the maximum pooling layer is to reduce the size of the input tensor to half of the original size.
2. The image-based generative adversarial neural network according to claim 1, characterized in that: The network is built based on the Tensorflow platform, and the optimizers all use the AdamOptimizer algorithm, which contains a learning rate parameter, a first momentum gradient descent factor, and a small constant; the summation operation uses the add algorithm; The convolution layer A, convolution layer B, convolution layer C, convolution layer D, convolution layer E, convolution layer F, convolution layer G, convolution layer H, convolution layer I, convolution layer J, convolution layer K, convolution layer L, convolution layer M, convolution layer N, convolution layer O, convolution layer P, convolution layer Q, convolution layer R, convolution layer S, convolution layer T, convolution layer U, convolution layer V, and convolution layer W have the same structure and function, and the construction parameters of their structure include input tensor, convolution kernel size, number of convolution kernels, step size of convolution kernel, filling method of convolution kernel, bias term of convolution kernel, and initial value of convolution kernel, and their function is to extract features of their input tensor; the convolution layers have different input tensors, convolution kernel sizes, and convolution kernels. The number, step size of the convolution kernel, and filling method of the convolution kernel are "SAME", "VALID", and "REFLECT". The initial value of the convolution kernel is a standard normal distribution with a standard deviation of 0.01, and the initial value is generated by the Initializers.Xavier_initializer algorithm. The initial value of the bias term of the convolution kernel is 0, and the initial value is generated by the Init_ops.Zeros_initializer algorithm. The convolution layer adopts the nn.conv2d algorithm, and its output is the sum of the convolution operation and the bias term, and the sum adopts the add algorithm. The construction parameters of the convolution layer A are: the size of the convolution kernel is 7*7, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer B are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer C are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer D are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1 , 2, 2, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer E are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer F are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer G are that the size of the convolution kernel is 7*7, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "SAME", and the number of convolution kernels is 3;The construction parameters of the convolution layer H are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer I are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 32; the construction parameters of the convolution layer J are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer K are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], and the number of convolution kernels is The filling method is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer L are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer M are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer N are that the size of the convolution kernel is 1*1, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 1; the construction parameters of the convolution layer O are that the size of the convolution kernel is 3* 3. The step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 64; the construction parameters of the convolution layer P are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 128; the construction parameters of the convolution layer Q are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 256; the construction parameters of the convolution layer R are that the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling method of the convolution kernel is "SAME", and the number of convolution kernels is 5 12; the construction parameters of the convolution layer S are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the top, bottom, left and right, and the number of convolution kernels is 16; the construction parameters of the convolution layer T are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right, and the number of convolution kernels is 32; the construction parameters of the convolution layer U are: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right, and the number of convolution kernels is 64;The construction parameters of the convolution layer V are as follows: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 2, 2, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the lower right corner, and the number of convolution kernels is 128; the construction parameters of the convolution layer W are as follows: the size of the convolution kernel is 3*3, the step size of the convolution kernel is [1, 1, 1, 1], the filling mode of the convolution kernel is "REFLECT", one unit is filled in the top, bottom, left and right corners, and the number of convolution kernels is 1; The convolutional layers 1, 2, 3 and 4 are convolutional kernels with specific requirements, whose convolutional kernel size is 3*3, whose step size is [1, 1, 1, 1], and whose filling mode is "VAILD"; the convolutional kernels of the convolutional layer 1 are [[[0, -1, 0], [0, 0, 0], [0, 1, 0]], [[0, -2, 0], [0, 0, 0], [0, 2, 0]], [[0, -1, 0], [0, 0, 0], [0, 1, 0]]], and the convolutional kernels of the convolutional layer 2 are [[ ... The convolution kernels are [[[0, 0, 0], [-1, 0, 1], [0, 0, 0]], [[0, 0, 0], [-2, 0, 2], [0, 0, 0]], [[0, 0, 0], [-1, 0, 1, [0, 0, 0]]], and the convolution kernels of convolution layer 3 are [[[0, -1, 0], [0, 2, 0], [0, -1, 0]], [[0, -2, 0], [0, 4, 0], [0, -2, 0]], [[0, -1, 0], [0, 2, 0], [0, 1, 0]]], The convolution kernel of convolution layer 4 is [[[0, 0, 0], [-1, 2, -1], [0, 0, 0]], [[0, 0, 0], [-2, 4, -2], [0, 0, 0]], [[0, 0, 0], [-1, 2, -1], [0, 0, 0]]]; the smoothing filter layer adopts the nn.depthwise_conv2d algorithm, and its construction parameters include the input tensor, the size of the convolution kernel, the step size of the convolution kernel, and the filling method of the convolution kernel. The size of the convolution kernel of the smoothing filter layer is 3 *3. The step size of the convolution kernel is [1, 1, 1, 1], and the filling mode of the convolution kernel is "SAME"; the initial value of the convolution kernel of the smoothing filter layer is a standard normal distribution with a standard deviation of 0.01, and the initial value is generated by the Initializers.Xavier_initializer algorithm; the initial value of the bias item of the convolution kernel of the smoothing filter layer is 0, and the initial value is generated by the Init_ops.Zeros_initializer algorithm; The color decomposition layer is based on the relation To achieve its function, R I , G I , B I R2, G2, and B3 are two inputs, which are called the three-channel data of input image I1 and input image I2. I , G I , B I is the data of the red, green and blue channels of the image I1, R2, G2 and B2 are the data of the red, green and blue channels of the image I2 respectively, R w , G w , B w are weight coefficients of red, green and blue channels respectively; the segmentation layer adopts Python.Skimage.Felzenszwalb segmentation algorithm, which divides the image into some small areas, each area has similar pixels and positions; the coloring layer colors each image area segmented by the segmentation layer, and if the standard deviation of the pixel values in the area is less than 20, the average color is used for coloring; if the standard deviation is between 20 and 40, the mixture of the average and median colors is used for coloring; if the standard deviation is greater than 40, the median color is used for coloring; The maximum pooling layer adopts the nn.max_pool algorithm, and the construction parameters of the maximum pooling layer include the input tensor, the size of the convolution kernel, the step size of the convolution kernel, and the filling method of the convolution kernel. The parameters of all the maximum pooling layers are that the size of the convolution kernel is 2*2, the step size of the convolution kernel is [1, 2, 2, 1], and the filling method of the convolution kernel is "SAME"; the normalization layer is implemented using the layer_norm algorithm; the compression layer adopts the clip_by_value algorithm, and its construction parameters include the input tensor and the compression range; the linear layer 1 adopts the shape and Image.Resize_bilinear algorithms, which expand the length and width of the input tensor by 2 times.
3. A method for generating cartoon face images based on image creation using a generative adversarial neural network, characterized in that: The following steps are involved: Step 1: Construct a real face color image dataset D1 and a cartoon face color image dataset D2 of size M*N; the images in the dataset D1 are real images collected by the device, and the images in the dataset D2 are cartoon images created by artists, which have clear contours, sparse color blocks, more details and other features, and soft strokes, more thick lines, smooth object surfaces, smooth colors in non-edge areas, and a soft overall image effect; Step 2: Select any pair of images from the data set D1 and the data set D2 in step 1, respectively referred to as input 1 and input 2, and then input the input 1 into the generation module described in claim 1 to obtain the output G1 of the generation module; Step 3: Input the output G1 obtained in step 2 and the input 1 into the enhancement module described in claims 1-2 to obtain the output G2 of the enhancement module; Step 4: Input the output G2 obtained in step 3 and the input 2 described in step 1 into the texture module of the cartoon style extraction module described in claims 1-2, and the texture module optimization controller of the texture module optimizes the texture module and the generation module according to its implicit output; input the output G2 obtained in step 3 and the input 2 described in step 1 into the surface module of the cartoon style extraction module described in claims 1-2, and the surface module optimization controller of the surface module optimizes the surface module and the generation module according to its implicit output; input the output G2 obtained in step 3 and the input 2 described in step 1 into the local texture module of the cartoon style extraction module described in claims 1-2, and the local texture module optimization controller of the local texture module optimizes the table local texture module and the generation module according to its implicit output; input G2 in step 3 and the input 1 described in step 1 into the content module of the real face extraction module described in claims 1-2, and the content module optimization controller of the content module optimizes the content module and the generation module according to its implicit output; The output G2 obtained in step 3 is input into the structural module of the real face extraction module described in claims 1-2, and the structural module optimization controller of the structural module optimizes the structural module and the generation module according to its implicit output; Step 5: Repeat steps 2-4 to train the network described in claims 1-2 for multiple times, and save the training results every KK∈[30, 1000] training times; after the total number of training times reaches iteration∈[50000, 200000] times, the final face cartoon style transfer model M is obtained; Step 6: The real face image to be processed is taken as input 1 and input into the model M obtained in step 5. The output G2 obtained by processing the generation module and enhancement module model in the model M is the high-quality cartoon-style face image quickly generated by the model M.
4. The method for generating cartoon face images based on image creation using a generative adversarial neural network according to claim 3, characterized in that: The method is implemented using Python language and Tensorflow platform; the learning rate parameter in claim 2 is 0.0002, the first momentum gradient descent factor is 0.5, and the small constant is 1*10 -8 The color decomposition layer described in claims 1-2 uses the Split module to separate the three channels of the input image I1 and the input image I2, and uses the Random.Uniform module to generate the random weight parameter R w , G w , B w ; The dataset D1 described in step 1 is 5000 images in the FFHQ dataset, and the dataset D2 is cartoon face images captured from movies produced by Kyoto Animation; the images in the datasets D1 and D2 described in step 1 are color images in JPG format with sizes M=256 and N=256; KK=500, iteration=100000 in step 5; the texture module, surface module, local texture module, content module and structure module described in step 4 involve λ1, λ2, λ3, λ4 and λ5 described in claim 1, preferably, λ1=10, λ2=1, λ3=300 and λ4=2*10 3 ,λ5=2*10 3 ; The cartoon-style face image obtained in step 6 is a high-quality cartoon face image of a specific cartoon style having the content features of input 1 described in step 1. The image has clear contours, more details, soft strokes, more thick lines, a smooth surface, smooth colors in non-edge areas, and the entire image is soft.
