Intelligent cartoon line manuscript generation system capable of keeping human face features

Abstract

The invention provides an intelligent cartoon line manuscript generation system capable of keeping human face features. The system comprises: 1, a backbone network, which is used for receiving cue word embedding representation input, controlling network embedding representation input, and generating a first draft of a cartoon line draft; the control network is used for receiving the human face, the human face key point embedded representation and the content image embedded representation, and generating a control network embedded representation; 3, four encoders of prompt words, human faces, human face key points and content images respectively generate corresponding embedded representations; and 4, a later-stage line draft adjustment module which performs local and overall style adjustment on the cartoon line draft generated by the backbone network by using a multi-layer generative adversarial network architecture to generate a final cartoon line draft finished product. According to the invention, the cartoon line manuscript with the specified face features can be efficiently generated, and the production efficiency of cartoon products is improved.

Description

Technical Field

[0001] The present invention belongs to the field of digital comics and generative artificial intelligence, and specifically relates to an intelligent digital comic line draft generation system that retains human face features. Background Art

[0002] Comic production requires plot setting, character setting, script writing, storyboard division, manuscript drawing, binding synthesis and other processes. The manuscript drawing stage generally requires manual drawing on manuscript paper or digital tablets. The workload of manuscript drawing is huge, and generally requires the main creator and assistants to work together to complete the sketch framework, line drawing and other steps.

[0003] The above creation mode has the following disadvantages

[0004] First, there is a lot of mechanical repetitive work and low efficiency. Line drawings require the main creators to have considerable artistic skills in advance and to manually draw the creative content. In the case of complex scenes, it is not even possible to produce a picture every day. A large number of repeated scenes or scenes with little change in the line drawings need to be repeatedly drawn, which is difficult to reuse. Secondly, it is difficult for multiple people to collaborate. The painting styles of multiple main creators cannot be unified, resulting in deviations in the artistic effects of the previous and subsequent characters and scenes. Third, due to the scarcity of high-level main creators, the facial makeup of characters in a large number of comics is monotonous and templated, which cannot correspond to the rich facial features of characters in the real world, affecting the artistic effect.

[0005] The existing intelligent comic generation methods do not solve the above problems well, especially the lack of methods to accurately and controllably draw facial features of characters and avoid stereotyped comic facial expressions.

[0006] In view of the above shortcomings, the present invention provides an intelligent comic line draft generation system. Summary of the invention

[0007] The purpose of the present invention is to solve the problem of low efficiency and inability to maintain style consistency in hand-drawn comic line drawings, and to solve the problem of accurately controlling the facial features of characters in the process of generating comic line drawings.

[0008] The intelligent comic line draft generation system for maintaining facial features provided by the present invention includes: a face embedding representation encoder and a face key point embedding representation encoder, which first acquire, manage, capture, segment, and arrange face calibration reference image materials, and then generate face information embedding representation and face key point information embedding representation. A prompt word embedding representation encoder is used to generate embedding representation for input text prompt words. A content image encoder is used to generate embedding representation for input content images. A control network is used to generate control network output based on input face information embedding representation and face key point information embedding representation, and content image embedding representation. A backbone network is used to control the backbone module based on input text prompt word embedding representation and control network output, so as to manipulate the backbone network and generate a preliminary line draft that meets content requirements and style requirements through white noise inverse diffusion. A post-line draft adjustment module is used to perform post-adjustment on the line draft generated by the backbone network to obtain the final comic line draft.

[0009] The intelligent comic line draft generation system for preserving facial features as described above is further preferably configured such that the facial embedding representation encoder and the facial key point embedding representation encoder have an image preprocessing function, and perform preprocessing operations such as cropping, correction, and enhancement on the facial reference image.

[0010] The face-preserving intelligent comic line draft generation system as described above is further preferably configured such that the face embedding representation encoder and the face key point embedding representation encoder utilize a pre-trained face recognition model to detect faces in a reference image and convert them into embedded representations.

[0011] In the above-mentioned intelligent comic line draft generation system with preserved facial features, it is further preferred that the prompt word embedding representation encoder converts the text prompt word input by the user into an embedding representation using a pre-trained model.

[0012] The intelligent comic line draft generation system with preserved facial features as described above is further preferably configured such that the content image encoder includes two steps: edge detection and embedding generation.

[0013] The intelligent comic line draft generation system for preserving facial features as described above is further preferably configured such that the edge detection step uses a Canny edge detection operator to detect significant edge features of the content image, and converts the original image into a binary image containing significant boundary or structural information.

[0014] The intelligent comic line draft generation system for preserving facial features as described above is further preferably configured such that the embedding generation step uses a convolutional neural network to perform feature extraction and multi-scale feature fusion on the binary image to obtain an embedded representation of the content image.

[0015] The intelligent comic line draft generation system for preserving facial features as described above is further preferably configured such that feature extraction converts low-level pixel information of the edge map into high-dimensional feature embedding containing semantic information through a convolutional neural network feature extractor.

[0016] The intelligent comic line draft generation system for preserving facial features as described above is further preferably configured such that, in the feature fusion operation, high-level features are gradually upsampled and then fused and added layer by layer with features from low layers, and the fused feature map contains multi-level information from detail edges to global structures.

[0017] The intelligent comic line draft generation system with facial feature preservation as described above is further preferably that the control network replicates the U-NET model structure in the backbone network and receives the aforementioned facial key point embedding representation, facial information embedding representation, and content image embedding representation as input. The feature map outputs of each layer of the U-NET decoder are connected to the backbone network module using a trainable zero convolution method.

[0018] The intelligent comic line draft generation system with preserved facial features as described above is further preferably configured such that in the zero convolution method connection operation, the initial weight of the convolution kernel is set to zero, and its weight is reset during the control network training process to ensure that the control network exerts an appropriate influence on the backbone network.

[0019] The intelligent comic line draft generation system with facial feature preservation as described above is further preferably that the backbone network includes: a U-NET model whose weights remain frozen during system training, whose input is a white noise image, and receives control of the control network output and the hint word embedding representation.

[0020] The intelligent comic line draft generation system with facial feature preservation as described above is further preferably configured such that the post-production line draft adjustment module includes: a semantic segmentation module, which uses a pre-trained model to semantically segment the line draft into multiple parts; and a generative adversarial network module, which uses a pre-trained generative adversarial network model to fine-tune the semantic parts of the segmented line drafts and finally combine and output them.

[0021] The intelligent comic line draft generation system with facial feature preservation as described above is further preferably: the generative adversarial network model in the post-line draft adjustment module includes two parts: a generator and a discriminator. The generator includes a global network, four local networks and a fusion network. The global network is responsible for overall adjustment, while the four local networks are responsible for detailed adjustment of the face, limbs, clothes, and image background. The discriminator includes a global discriminator and four local discriminators, which jointly identify whether the adjusted line draft output by the generator is true.

[0022] The present invention also discloses an intelligent comic line draft generation method for maintaining facial features. The method comprises the following steps: Step 1: pre-training a generative adversarial network fine-tuning model. Step 2: preparing a backbone network module and a control network module. Step 3: preparing a face reference image and a content image. Step 4: inputting the prepared face reference image into a face embedding representation encoder and a face feature point embedding representation encoder to obtain two types of embedding representations. The content image is pre-processed by edge detection and the like, and inputted into a content image encoder to obtain a content image embedding representation. Step 5: inputting the above embedding representations into the control network module respectively. Step 6: inputting a text prompt word into a prompt word encoder to obtain a prompt word embedding representation. Step 7: inputting the embedding representation outputted by the prompt word encoder into the backbone network. Step 8: the output of the control network generation module is linked to the backbone network generation module through a pre-trained zero convolution weight. Step 9: inputting the line draft draft outputted by the backbone network into a later line draft adjustment module, and the final comic line draft result is obtained after adjustment.

[0023] The above-mentioned intelligent line draft generation method for maintaining facial features is further preferably that step 1 specifically includes: step 1.1 preparing paired actual pictures and corresponding hand-drawn comic line draft picture training data. This step requires a person with a certain artistic background to draw a comic line draft image according to the selected actual picture. Step 1.2 Using the defined network model structure and loss function, the network is trained through paired data.

[0024] The pre-prepared paired content images and line drawings are used to train the generative adversarial network model in the post-line drawing adjustment module. The discriminator loss function used to train the generative adversarial network is

[0025] The point loss function used in training the generative adversarial network fine-tuning model is

[0026] The local loss used to train the generative adversarial network is the sum of the L-2 norm distances between the four local regions generated by the local network and the four local regions of the manually drawn comic reference image.

[0027] In the above-mentioned intelligent comic line draft generation method with preserved facial features, step 2 specifically includes: step 2.1 copying the U-NET module structure of the backbone network. Step 2.2 training the zero convolution module that controls the network weight and connects to the backbone network.

[0028] In the above-mentioned intelligent comic line draft generation method with preserved facial features, step 3 specifically includes: step 3.1 obtaining image data containing the target face. Open source image datasets, real face images on the Internet, etc. can be used. Step 3.2 performing operations such as cropping, rotating, and enhancing the image to regularize it.

[0029] In the above-mentioned intelligent comic line draft generation method with preserved facial features, the face reference image embedding representation encoder and the face key point embedding representation encoder in step 4 adopt an open source face information processing model. The content image embedding representation encoder adopts an open source pre-trained model.

[0030] In the above-mentioned intelligent comic line draft generation method that preserves facial features, the prompt word encoder in step 7 adopts the model completed by open source training in SDXL.

[0031] In the above-mentioned intelligent comic line draft generation method with facial feature preservation, step 9 specifically includes: step 9.1 using the open source segmentation model to semantically segment the comic line draft sketch output by the backbone network into four components. Step 9.2 using the four local networks of the generator to fine-tune the four semantic parts after segmentation respectively. Step 9.3 using the outputs of the four local networks of the generator to assemble and splice into a complete comic line draft image and output it.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] The invention belongs to the field of generative artificial intelligence. By encoding the face in the face reference image as the face information embedding representation and the face key point embedding representation, the facial feature information in the face reference image is obtained. By freezing the weight of the backbone generation network, the huge overhead in training is avoided. At the same time, freezing the weight of the backbone network can maintain the powerful generation ability of the original model. The feature maps of each layer output by the trainable control network are connected to the feature maps of each layer of the backbone network through the trainable zero convolution method, so as to minimize the impact on the generation ability of the backbone network. While receiving the face embedding representation information and the face key point embedding representation information, the control network also receives the embedding representation of the content image at the same time and connects it to the backbone network. The backbone network is controlled to generate an image that maintains the facial features of the face, the content is consistent with the content image, and the style is a comic line draft. The invention uses a trained hierarchical generative adversarial network model to further adjust the comic line draft output by the backbone network. Through semantic segmentation, the comic line draft output by the backbone network is decomposed into four different parts, and the style is adjusted and improved by using the generator local networks that are trained separately. Finally, the different parts are combined and output. Compared with the existing methods, the present invention realizes the content preservation and facial feature specification when intelligently generating comic line drawings. The present invention realizes the output of different parts of comic line drawings in different artistic styles, thereby improving efficiency and output quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of an intelligent comic line draft generation system for preserving facial features provided in Embodiment 1 of the present invention.

[0035] Figure 2 This is a structural block diagram of a generator of a post-production line draft adjustment module of an intelligent comic line draft generation system for preserving facial features provided in Embodiment 1 of the present invention.

[0036] Figure 3 This is a structural block diagram of a discriminator of a post-production line draft adjustment module of an intelligent comic line draft generation system for preserving facial features provided in the first embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of the present application.

[0038] As the background technology, the current comic line draft generation industry has serious inefficiencies in manual hand-drawing. Existing intelligent comic generation technologies have problems such as the inability to accurately control the style and the stereotyped output of characters.

[0039] In response to the problems arising in the related art, an embodiment of the present application provides an intelligent comic line draft generation method that preserves facial features.

[0040] like Figure 1 As shown, the present invention realizes a new intelligent comic line draft generation pipeline (Pipeline), which is a wrapper of multiple modules and methods used in the present invention.

[0041] Specifically, the pipeline system encapsulates a backbone generation network module based on a diffusion network. A control network module with the same structure as the backbone generation network but with trainable weights. A post-production line draft adjustment module based on a generative adversarial network, as well as a face embedding representation encoder, a face key point embedding representation encoder, a content image embedding representation encoder, and a prompt word embedding representation encoder.

[0042] The implementation of the present invention includes a face embedding representation encoder and a face key point embedding representation encoder. The encoder group first performs cropping, rotation, and enhancement operations on the reference image, so that the model can better cope with situations such as camera angle changes, image offset, or partial occlusion of the object.

[0043] Cropping is to select an area from the original image and output the area as a new image. The main purpose of cropping is to remove unnecessary edges or noise, or to focus on a certain area in the image. The steps of cropping are: first determine the starting point of cropping and the width and height of the cropping window The clipping end point is calculated as

[0044] After determining the starting point, width, height and end point, the corresponding area in the image is extracted as the cropping result.

[0045] Rotation is a geometric transformation that rotates an image around the center of the image at a certain angle. The rotation matrix is:

[0046] Where θ is the rotation angle (in radians), a positive angle indicates counterclockwise rotation, and a negative angle indicates clockwise rotation. The specific implementation steps are to first determine the rotation angle, then generate the rotation transformation angle, and then apply the rotation matrix to transform each pixel coordinate of the image.

[0047] Scaling is to enlarge or reduce an image by a certain ratio. The scaling transformation matrix is:

[0048] in, and Represents the scaling factor on the X-axis and Y-axis respectively. The specific implementation steps are to first determine the scaling factor, then generate a scaling transformation matrix, apply the scaling matrix to each pixel of the image, and adjust the pixel coordinates.

[0049] After implementing the above preprocessing operations, the encoder group loads the pre-trained model and configuration environment.

[0050] After loading the pre-trained model and configuration environment, the encoder group first performs face detection, returns the face bounding box and obtains face key points (landmarks) for face alignment and subsequent feature extraction. Through affine transformation, the detected key points are aligned to the standard face position.

[0051] The encoder group maps aligned face images into fixed-length embedding representations through a pre-trained deep convolutional neural network.

[0052] After obtaining the facial key points, the encoder group aligns their coordinates to the standard template and maps the aligned key point coordinates into a fixed-length embedding representation through a pre-trained deep convolutional neural network.

[0053] The implementation of the present invention includes a content image encoder. The specific implementation includes: step 1, performing the aforementioned cropping, rotation, and enhancement operations on the content image. Step 2, generating posture control information. Step 3, generating depth control information. Step 4, generating edge detection information. Step 5, inputting the aforementioned posture control information, depth control information, and edge detection information into a pre-trained deep convolutional neural network to generate a fixed-length content image embedding representation code.

[0054] The implementation of the present invention includes a control network module encapsulated in a pipeline. Its specific implementation includes: Step 1, constructing a network model with the same structure as the backbone generation network U-NET, whose weights are different from those of the backbone network and can be trained and changed. Step 2, embedding the aforementioned face image into the identification encoder, embedding the face key points into the representation encoder, and embedding the output of the content image encoder into the representation input control network U-NET model. Step 3, accumulating the output feature map of each layer encoder of the control network U-NET to the corresponding layer output feature map of the backbone network U-NET decoder through a trainable zero convolution method, thereby realizing the control of the comic line draft output by the backbone network.

[0055] The implementation of the present invention includes a backbone network module encapsulated in a pipeline. The specific implementation includes: step 1, loading a pre-trained benchmark diffusion model. Step 2, setting target style prompt words and negative prompt words for the pre-trained benchmark model. Step 3, embedding the white noise image and the prompt words into the encoder output, and controlling the network output combination to input the backbone network module to obtain the comic line draft output corresponding to the content image.

[0056] The implementation of the present invention includes a post-production line draft adjustment module. The specific implementation includes: step 1, constructing a semantic-based segmenter. Step 2, constructing a generator. Step 3, constructing a discriminator. Step 4, jointly training the generator and discriminator networks using a specific loss function. Step 5, using the trained generator network to generate a post-production adjusted comic line draft final draft.

[0057] Implementation step 1 of the post-production line drawing adjustment module: Use the pre-trained semantic-based image segmenter to divide the input line drawing into four components: face, limbs, clothes, and background.

[0058] Step 2 of the post-production line draft adjustment module: includes constructing a global network (Global Net): used to process the overall features of the input image. This global network includes a four-layer pyramid structure encoder neural network, four residual network middle layers, and a four-layer pyramid structure decoder neural network. After training, this global network can extract the overall features of the input comic line draft.

[0059] The post-production line draft adjustment module implements step 2, including constructing four local networks (Local Nets): used to perform fine-grained feature extraction and classification of specific local areas in the input image. These four local networks each contain a two-layer pyramid encoder neural network, three residual network middle layers, and a two-layer pyramid decoder neural network. After these four local networks are trained separately, they can perform feature extraction and adjustment on the four different parts of the input line draft, namely the aforementioned face, limbs, clothes, and background.

[0060] The post-production line draft adjustment module implements step 2, including constructing a fusion network (Fusion Net): In the final stage of the generator, the outputs of the global network and the local network are combined in the fusion network to generate the final output image. The fusion network has the same structure as the global network, receiving the outputs of the global network and the local network and merging them into a complete comic line draft.

[0061] The post-production line draft adjustment module implements step 3, including constructing a global discriminant network (Global Net): This network uses a pyramid convolutional neural network similar to the global network encoder in the generator above, but the difference is that the dimension of the final layer is different. Its final layer is a 64x64-dimensional convolutional neural network, which is equivalent to dividing the input image to be judged into 64x64 small pieces. The final layer judges the authenticity of each small piece of image, and calculates the arithmetic average to judge the authenticity of the line draft generated by the overall generator.

[0062] The post-production line draft adjustment module implements step 3, including constructing a local discriminant network (Local Nets): This network uses a pyramid convolutional neural network similar to the local network encoder in the generator above, but the difference is that the dimension of the final layer is different. Suppose the local area after semantic segmentation is rectangle, the dimensions of the final layers of the four local networks are , its discrimination mechanism is the same as the aforementioned global discrimination network.

[0063] The post-production line draft adjustment module implements step 3, and the discriminator outputs the judgment result to ensure that the adjusted line draft image generated by the generator is not only consistent with the backbone network input line draft image in terms of overall structure, but also achieves a high degree of authenticity in local details.

[0064] In the implementation step 4 of the post-production line draft adjustment module, the loss function used for parallel training of the generator and discriminator networks consists of three parts: the first is adversarial loss, the second is point loss, and the third is local loss; The adversarial loss is: ; in Indicates traversing all training image pairs Take the arithmetic mean of the loss values. Here is the first draft of the comic line drawing generated by the backbone network described in claim 3, and This is the final draft of the corresponding comic line drawing drawn manually. The global discriminator and the local discriminator as described in claim 7; The point loss is: ; This loss represents the point-by-point difference between the fine-tuned line drawing generated by the generator and the final line drawing of the corresponding comic line drawing drawn manually. The local loss is the sum of the L-2 norm distances between the local areas generated by the four local networks and the four local areas of the manually drawn comic reference image.

[0065] The post-production line draft adjustment module implements step 5. The comic line draft generated by the aforementioned backbone network is first processed by the global network to generate its global features. At the same time, the local network extracts the key areas of the face from the input image and performs fine-grained feature extraction on these areas. The global and local features are finally merged through the fusion network to generate the final output comic line draft.

[0066] This embodiment also provides an electronic terminal, including a processor and a memory.

[0067] The memory is used to store a computer program, and the processor is used to execute the stored computer program, so that the terminal executes any one of the methods in this embodiment.

Claims

1. An intelligent comic line drawing generation system that preserves facial features. It is characterized by: include: The backbone network U-NET module is used to embed the input representation according to the prompt word, control the network U-NET output, and generate the first draft of the comic line through the diffusion algorithm; The control network U-NET module is used to receive the encoded embedding representation of the face reference image and the content image, generate the output representation, connect to the backbone network through zero convolution, and control the comic line generation of the backbone network; The post-production line draft adjustment module is used to fine-tune the initial draft of the comic line draft in parts, further improve the style, and obtain the final line draft; The prompt word embedding representation encoder, the face embedding representation encoder, the face key point embedding representation encoder, and the content image encoder respectively realize the encoding from the relevant input data to the embedding representation acceptable to the model.

2. The intelligent comic line draft generation system with facial feature preservation according to claim 1, characterized in that: The face embedding representation encoder and the face key point embedding representation encoder include three functions: preprocessing of face reference images, encoding of face regions and encoding of face key points.

3. The intelligent comic line draft generation system with facial feature preservation according to claim 1, characterized in that: The backbone network module is a pre-trained, weight-frozen U-NET architecture deep neural network, which can generate a target style comic line draft by receiving the cue word embedding representation coupled with a multi-head attention mechanism and the control network output vector connected in a zero-convolution manner.

4. The intelligent comic line draft generation system with facial feature preservation according to claim 1, characterized in that: The control network module is a U-NET architecture deep neural network with the same structure as the backbone network module, but with trainable weights. This control network simultaneously receives the reference face image embedding representation, the reference face key point embedding representation, and the content image embedding representation to generate output; The output of the control network is accumulated to the feature map of the corresponding layer of the backbone network through the zero convolution mechanism to control the output behavior of the backbone network.

5. The intelligent comic line draft generation system with facial feature preservation according to claim 1, characterized in that: The post-production line draft adjustment module includes a semantic segmentation module and a pre-trained generative adversarial network structure style adjustment module; The semantic segmentation module uses a pre-trained segmentation model to semantically segment the output line drawing of the backbone network module as claimed in claim 3 into four separate areas: facial organs, limbs, clothes, and image background, and perform style adjustments on each of them; The generator in the generative adversarial network includes a global network, four local networks and a fusion network; The global network is a pre-trained U-NET architecture deep neural network, which is used to adjust the output line draft of the backbone network module described in claim 3 to generate an adjusted overall architecture; The local network is four pre-trained smaller-scale U-NET architecture deep neural networks, which are used to adjust the style of the facial organs, limbs, clothes, and image background of the output line draft of the backbone network module of claim 3; The fusion network is used to fuse the outputs of the global network and the four local networks into a complete comic line drawing image.

6. The discriminator in the generative adversarial network structure according to claim 6 comprises a global discriminator and four local discriminators; The global discriminator is a pre-trained pyramid network, and the four local discriminators are four smaller pyramid networks; The global discriminator and the local discriminator jointly identify whether the adjusted comic line drawing output by the generator is real.

7. The generative adversarial network discriminator of claim 7, wherein: The loss function required for training consists of three parts: the first is adversarial loss, the second is point loss, and the third is local loss; The adversarial loss is: ； in Indicates traversing all training image pairs Take the arithmetic mean of the loss values. Here is the first draft of the comic line drawing generated by the backbone network described in claim 3, and This is the final draft of the corresponding comic line drawing drawn manually. The global discriminator and the local discriminator as described in claim 7; The point loss is: ； This loss represents the point-by-point difference between the fine-tuned line drawing generated by the generator and the final line drawing of the corresponding comic line drawing drawn manually. The local loss is the sum of the L-2 norm distances between the local areas generated by the four local networks and the four local areas of the manually drawn comic reference image.

8. An electronic terminal, characterized in that: include: A processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes the method according to any one of claims 1 to 7.

Images