Face image generation method, device, equipment and storage medium

Abstract

The present application provides a kind of face image generation method, device, equipment and storage medium, wherein the method comprises: obtaining input sketch and face attribute;The sketch is input into the preset face generation model, the token sequence of each partition in the sketch is calculated by the face generation model, and the global attention of the sketch is calculated based on the token sequence;According to the global attention, the image feature of the sketch is calculated;The image feature is mapped to face feature space, and the image feature mapped to face feature space is edited, and the corresponding face feature of the sketch is obtained;According to the face feature, the corresponding face image is generated.In this way, by calculating the global attention of the sketch, high-quality image features are extracted, the requirement for the quality of the sketch is reduced, high-quality facial images can be generated from rough or even incomplete freehand sketches, and the editable range of the generated face is improved through the feature editing function.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a face image generation method, device, equipment and storage medium. BACKGROUND

[0002] With the continuous development of the game market and the growing of the young user demand, the requirement of art quality is also increasing. Usually, an artist needs rich art experience to design a character image, and a large amount of reference and design, and is subject to copyright constraints. Therefore, it is urgent to provide a low-drawing threshold high-definition face image generation method.

[0003] A natural and intuitive method for generating a face image with a low drawing threshold is to draw a sketch. Sketch is an effective visual thinking form. It describes the structure and shape of the target object, and is more specific than text description. Converting a sketch into a real image has attracted wide attention in the field of computer and computer graphics. In the prior art, image mapping is mainly used to generate a face image based on a sketch.

[0004] Then, the image mapping cannot obtain a clear and natural picture when the input sketch only contains part of the structure or the output sketch is missing. The main reason is that this kind of method strictly follows the lines of the input sketch for image synthesis, so they will not generate new structures in areas with few or no lines. The face is composed of parts with clear structures, such as nose, mouth, eyes, etc. Even if the input sketch does not have these edges at the corresponding position, the synthesis should still maintain these structures to generate a face with realistic feeling, that is, the quality of the sketch required by the sketch to generate a face image through image mapping is high, and it is impossible to generate a face image from a low-quality sketch. SUMMARY

[0005] The main purpose of the present application is to solve the technical problem that the existing sketch cannot generate a face image using a low-quality sketch.

[0006] The first aspect of the present application provides a face image generation method, which comprises: obtaining an input sketch and face attributes;

[0007] inputting the sketch into a preset face generation model, calculating the token sequence of each partition in the sketch through the face generation model, and calculating the global attention of the sketch based on the token sequence; calculating the image feature of the sketch according to the global attention; mapping the image feature to the face feature space, and performing feature editing on the image feature mapped to the face feature space to obtain the corresponding face feature of the sketch; and generating a corresponding face image according to the face feature.

[0008] Optionally, in the first implementation manner of the first aspect of the present application, the face generation model comprises a feature extraction network, a feature editing network and a face generation network.

[0009] Optionally, in the second implementation manner of the first aspect of the present application, the feature extraction network is composed of M stages and N style blocks, the stage is composed of a Patch embedding layer and a Transformer Encoder, and M and N are both natural numbers greater than 1; the step of inputting the sketch into the preset face generation model, calculating the token sequence of each partition in the sketch through the face generation model, and calculating the global attention of the sketch based on the token sequence comprises: inputting the sketch into the preset face generation model, calculating the token sequence of the corresponding partition in the sketch through the Patch embedding layer of the M stages in the feature extraction network; and calculating the global attention of the sketch according to the token sequence through the Transformer Encoder in the stage.

[0010] Optionally, in the third implementation manner of the first aspect of the present application, the feature editing network is composed of a clip model and a fully connected layer; and the step of mapping the image feature to the face feature space, editing the image feature mapped to the face feature space, and obtaining the corresponding face feature of the sketch comprises: mapping the image feature to the face feature space; extracting the text feature information of the face attribute through the clip model in the feature editing network; mapping the text feature information to the face feature space through the fully connected layer in the feature editing network, and obtaining the corresponding face feature of the sketch according to the image feature and the text feature information in the face feature space.

[0011] Optionally, in the fourth implementation manner of the first aspect of the present application, the step of generating the corresponding face image according to the face feature comprises: obtaining at least one random noise, and generating at least one face image based on the at least one random noise and the face feature through the face generation network.

[0012] Optionally, in a fifth implementation form of the first aspect of the present application, the face generation network is composed of a plurality of generation modules, each of which is composed of a feature layer and an up-sampling layer; the generating a face image based on the random noise and the face feature through the face generation network comprises: superimposing the random noise into the feature layer of each generation module through weight demodulation; inputting a preset fixed input into the feature layer in the first generation module and adding the face feature into the feature layer through adaptive normalization to generate a corresponding face feature image, and inputting the processed face feature image through the corresponding up-sampling layer as the input of the next generation module; repeating the face feature image generation step until the output of the last generation module, and taking the face feature image output by the last generation module as the face image.

[0013] Optionally, in a sixth implementation form of the first aspect of the present application, before the input sketch is obtained and input into the preset face generation model, the method further comprises: obtaining a sample face image and pre-processing the sample face image to obtain a sample sketch; inputting the sample sketch into a preset neural network model to obtain a model generated image; calculating a preset loss function according to the model generated image and the sample face image to obtain a corresponding loss function value; determining whether the loss function value is less than a preset loss threshold; if not, performing back propagation on the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; and if yes, constructing a face generation model according to the network parameters of the neural network model.

[0014] Optionally, in a seventh implementation form of the first aspect of the present application, the obtaining a sample face image and pre-processing the sample face image to obtain a sample sketch comprises: obtaining a sample face image and processing the sample face image using a preset filter to obtain a preliminary sketch; simplifying the preliminary sketch to obtain a simplified sketch; and performing sample augmentation on the simplified sketch to obtain a sample sketch.

[0015] Optionally, in an eighth implementation form of the first aspect of the present application, the simplifying the preliminary sketch to obtain a simplified sketch comprises: extracting edges of the preliminary sketch using a canny operator; identifying all connected edges in the edges using a fincounters function; calculating edge lengths of the connected edges and removing connected edges with edge lengths less than a preset length threshold; and generating a simplified sketch according to the remaining connected edges after the removal.

[0016] The second aspect of the present application provides a face image generation device, comprising: an acquisition module configured to acquire an input sketch and face attributes; an attention calculation module configured to input the sketch into a preset face generation model, calculate token sequences of each partition in the sketch through the face generation model, and calculate global attention of the sketch based on the token sequences; a feature calculation module configured to calculate image features of the sketch according to the global attention; a mapping module configured to map the image features to a face feature space, and perform feature editing on the image features mapped to the face feature space to obtain corresponding face features of the sketch; and a face generation module configured to generate a corresponding face image according to the face features.

[0017] Optionally, in the first implementation manner of the second aspect of the present application, the face generation model comprises a feature extraction network, a feature editing network and a face generation network.

[0018] Optionally, in the second implementation manner of the second aspect of the present application, the feature extraction network is composed of M stages and N style blocks, the stage is composed of a Patch embedding layer and a Transformer Encoder, and M and N are both natural numbers greater than 1; the attention calculation module is specifically configured to: input the sketch into the preset face generation model, calculate token sequences of corresponding partitions in the sketch through the Patch embedding layer of the M stages in the feature extraction network; and calculate global attention of the sketch according to the token sequences through the Transformer Encoder in the stage.

[0019] Optionally, in the third implementation manner of the second aspect of the present application, the mapping of the image features to the face feature space and the feature editing on the image features mapped to the face feature space to obtain the corresponding face features of the sketch comprise: mapping the image features to the face feature space; extracting text feature information of the face attributes through a clip model in the feature editing network; mapping the text feature information to the face feature space through a full connection layer in the feature editing network, and obtaining the corresponding face features of the sketch according to the image features and the text feature information in the face feature space.

[0020] Optionally, in the fourth implementation manner of the second aspect of the present application, the face generation module is specifically configured to: acquire at least one random noise, and generate at least one face image based on the at least one random noise and the face features through the face generation network.

[0021] Optionally, in a fifth implementation form of the second aspect of the present application, the face generation network is composed of a plurality of generation modules, each generation module being composed of a feature layer and an up-sampling layer; the face generation module is further configured to: superimpose the random noise into the feature layer of each generation module by means of weight demodulation; input a preset fixed input into the feature layer in the first generation module, and add the face feature into the feature layer by means of adaptive normalization to generate a corresponding face feature image, which is input into the corresponding up-sampling layer for processing and then used as the input of the next generation module; repeat the face feature image generation step until the output of the last generation module, and use the face feature image output by the last generation module as the face image.

[0022] Optionally, in a sixth implementation form of the second aspect of the present application, the face generation device further comprises a model training module, which specifically comprises: a preprocessing unit configured to obtain a sample face image and pre-process the sample face image to obtain a sample sketch; an input unit configured to input the sample sketch into a preset neural network model to obtain a model generated image; a loss calculation unit configured to calculate a preset loss function according to the model generated image and the sample face image to obtain a corresponding loss function value, and determine whether the loss function value is less than a preset loss threshold; a back propagation unit configured to, if the loss function value is not less than the preset loss threshold, perform back propagation on the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; and a model generation unit configured to, if the loss function value is less than the preset loss threshold, construct a face generation model according to the network parameters of the neural network model.

[0023] Optionally, in a seventh implementation form of the second aspect of the present application, the preprocessing unit is specifically configured to: obtain a sample face image and process the sample face image using a preset filter to obtain a preliminary sketch; perform simplification processing on the preliminary sketch to obtain a simplified sketch; and perform sample augmentation on the simplified sketch to obtain a sample sketch.

[0024] Optionally, in an eighth implementation form of the second aspect of the present application, the preprocessing unit is specifically further configured to: extract edges of the preliminary sketch using a canny operator; identify all connected edges in the edges using a fincounters function; calculate edge lengths of the connected edges, and remove the connected edges with edge lengths less than a preset length threshold; and generate a simplified sketch according to the connected edges remaining after the removal.

[0025] The third aspect of the present application provides a face image generation device, comprising: a memory and at least one processor, the memory storing instructions, and the memory and the at least one processor being interconnected by a circuit; the at least one processor invokes the instructions in the memory to enable the face image generation device to perform the steps of the face image generation method described above.

[0026] The fourth aspect of the present application provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to perform the steps of the face image generation method described above.

[0027] The face image generation method, device, equipment and storage medium described above, by obtaining the input sketch and face attribute; inputting the sketch into a preset face generation model, calculating the token sequence of each partition in the sketch through the face generation model, and calculating the global attention of the sketch based on the token sequence; calculating the image feature of the sketch according to the global attention; mapping the image feature to the face feature space, and performing feature editing on the image feature mapped to the face feature space to obtain the corresponding face feature of the sketch; and generating the corresponding face image according to the face feature. In this way, the global attention of the sketch is calculated to realize high-quality extraction of image features, reduce the requirement for the quality of the sketch, generate a high-quality face image from a rough or even incomplete freehand sketch, and improve the editable range of the generated face through the feature editing function.

[0028] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description, claims and drawings.

[0029] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The first embodiment of the face image generation method in the embodiments of the present application is shown in the figure;

[0031] Figure 2 The first embodiment of the face image generation method in the embodiments of the present application is shown in the figure;

[0032] Figure 3 The first embodiment of the face image generation method in the embodiments of the present application is shown in the figure;

[0033] Figure 4An embodiment diagram of a face image generation device in the embodiments of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application provide a face image generation method, device, equipment and storage medium, the face image generation method comprising the following steps: obtaining an input sketch and face attributes; inputting the sketch into a preset face generation model, calculating a token sequence of each partition in the sketch through the face generation model, and calculating a global attention of the sketch based on the token sequence; calculating an image feature of the sketch according to the global attention; mapping the image feature to a face feature space, and performing feature editing on the image feature mapped to the face feature space to obtain a corresponding face feature of the sketch; and generating a corresponding face image according to the face feature. In the method, the global attention of the sketch is calculated to realize high-quality extraction of the image feature, the requirement for the quality of the sketch is reduced, a high-quality face image can be generated from a rough or even incomplete freehand sketch, and the editable range of the generated face is improved through the feature editing function.

[0035] The terms "first", "second", "third", "fourth" and the like in the description, claims, as well as throughout the tables and figures of the disclosure, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments described herein are, for example, capable of proper

[0036] For the convenience of understanding, the specific flow of the embodiments of the present application is described below. Please refer to Figure 1 The first embodiment of the face image generation method in the embodiments of the present application comprises the following steps:

[0037] 101. Obtain an input sketch and face attributes;

[0038] It can be understood that the execution subject of the present application can be a face image generation device, and can also be a terminal or a server, and the specific execution subject is not limited herein. The embodiments of the present application take the server as the execution subject for example.

[0039] In this embodiment, the user first manually sketches a draft. The quality of the sketch does not need to be too high. The user then inputs the facial attributes (more than thirty types) of the desired generated face image, such as race, skin color, hair color, age, gender, emotion, and facial features. The user can adjust and edit the facial attributes of the generated face in real time. For example, races include yellow, white, and black people, and hair colors include black, yellow, and red. By setting different facial attributes and combining them with the user's sketch, a face image that matches the sketch and facial attributes is finally generated.

[0040] 102. Input the sketch into the preset face generation model, calculate the token sequence of each partition in the sketch through the face generation model, and calculate the global attention of the sketch based on the token sequence;

[0041] In this embodiment, a face generation model is pre-trained. Specifically, the face generation model includes a feature extraction network, a feature editing network, and a face generation network. The main function of the feature extraction network is to extract feature information from the sketch. The main function of the feature editing network is to expand the semantic information of the face attributes and embed it into the feature information of the sketch to obtain face features. The main function of the face generation network is to convert the extracted face features into a real image.

[0042] Specifically, in this embodiment, the feature extraction network consists of M stages and N style blocks. Each stage comprises a patch embedding layer and a Transformer Encoder, where M and N are both natural numbers greater than 1. The step of inputting the sketch into a preset face generation model, calculating the token sequence of each partition in the sketch through the face generation model, and calculating the global attention of the sketch based on the token sequence includes: inputting the sketch into the preset face generation model, calculating the token sequence of the corresponding partition in the sketch through the patch embedding layers of the M stages in the feature extraction network, and calculating the global attention of the sketch based on the token sequence through the Transformer Encoder in each stage.

[0043] In this embodiment, M is 4 and N is 18. In the feature extraction network, each stage consists of one patch embedding layer and two Transformer Encoders. The role of patch embedding is to transform the output of the previous stage into a token sequence, preparing for the subsequent attention mechanism. Each token actually corresponds to a partition, which intuitively reduces the size of the feature space. Each Transformer Encoder actually calculates global attention on the token sequence.

[0044] In this embodiment, the face generation model is obtained through the following steps: acquiring sample face images and preprocessing the sample face images to obtain sample sketches; inputting the sample sketches into a preset neural network model to obtain model-generated images; calculating a preset loss function based on the model-generated images and the sample face images to obtain corresponding loss function values; determining whether the loss function value is less than a preset loss threshold; if not, backpropagating the loss function value to update the network parameters of the neural network model until the obtained loss function value is less than the preset threshold; if yes, constructing a face generation model based on the network parameters of the neural network model.

[0045] Specifically, the step of acquiring sample face images and preprocessing the sample face images to obtain sample sketches includes: acquiring sample face images and processing the sample face images using a preset filter to obtain preliminary sketches; simplifying the preliminary sketches to obtain simplified sketches; and augmenting the simplified sketches to obtain sample sketches.

[0046] The sample face images refer to a dataset containing face images. In this embodiment, the CelebA-HQ high-definition face dataset is mainly used as the training and testing dataset, which includes a total of 30k images, each with a resolution of 1024*1024. To ensure the authenticity and diversity of the experimental data, the dataset is divided, with 80% of the face data used as the training set and 20% used as the testing set.

[0047] In this embodiment, the sample face image needs to be preprocessed to obtain the corresponding matching sketch. Specifically, the face image is input into the image software, and the sample face image is processed by the filter in the image software. In this embodiment, the filter used is a filter that converts the sample face image into a pencil sketch. During the filter processing, most of the facial details of the original image are preserved while the remaining noise is removed to obtain the preliminary sketch.

[0048] Furthermore, the simplification process of the preliminary sketch to obtain a simplified sketch includes: extracting the edges of the preliminary sketch using the Canny operator; identifying all connected edges in the edges using the Fincounters function; calculating the edge length of each connected edge and removing connected edges whose edge length is less than a preset length threshold; and generating a simplified sketch based on the remaining connected edges after removal.

[0049] In this embodiment, the preliminary sketch still retains a large amount of detail and needs further simplification. Therefore, a preset sketch simplification model is used to simplify the preliminary sketch, enabling the detection and extraction of edge components using the Canny operator. The Canny operator performs edge detection as follows: smoothing the image using a Gaussian filter, calculating the gradient magnitude and gradient angle images, applying non-maximum suppression to the gradient magnitude image, and using double thresholding and connectivity to detect and connect edges. After extracting the edges, the fincounters function is used to identify all connected edges within the edges. The fincounters function is a contour detection function that can identify and extract connected edges. After extracting the connected edges of the preliminary sketch, a fixed threshold is used to remove small edges, further simplifying the preliminary sketch to obtain a simplified sketch.

[0050] In practical applications, the diverse drawing styles of different individuals lead to significant variations in the distribution of input sketches, posing a substantial challenge to the generalization ability of the model. This invention employs JOJOGAN for few-shot augmentation of sketches, generating a large number of images in the same style using a small amount of data as a reference. JOJOGAN is a one-shot style transfer model that uses a GAN to invert the reference style image into approximate paired training data, generating realistic face images based on the stylization code and matching them with the reference style image. This dataset is then used to fine-tune StyleGAN, and new input images can be used. JOJOGAN will invert the input image to convert it to the specific style, thus simplifying the sketch augmentation under different styles and obtaining sample images of various styles.

[0051] 103. Calculate the image features of the sketch based on global attention;

[0052] In this embodiment, each stage in the feature extraction network consists of one patch embedding layer and two Transformer Encoders. The purpose of patch embedding is to transform the output of the previous stage into a token sequence, preparing for the subsequent attention mechanism. Each token actually corresponds to a partition, which intuitively reduces the size of the feature space. Each Transformer Encoder actually calculates global attention on the token sequence. Each style block consists of several 3*3 convolutions and a fully connected layer. After each convolution, a downsampling operation is performed, and the calculation is performed through the downsampling operation and the fully connected layer.

[0053] 104. Map the image features to the face feature space, and perform feature editing on the image features mapped to the face feature space to obtain the face features corresponding to the sketch;

[0054] In this embodiment, the feature editing network consists of a clip model and fully connected layers. The step of mapping the image features to a face feature space and then performing feature editing on the image features mapped to the face feature space to obtain the corresponding face features of the sketch includes: mapping the image features to a face feature space; extracting text feature information of the face attributes through the clip model in the feature editing network; mapping the text feature information to the face feature space through the fully connected layers in the feature editing network, and obtaining the corresponding face features of the sketch based on the image features and text feature information in the face feature space.

[0055] Specifically, the CLIP model uses text as a supervisory signal to train a transferable visual model. CLIP stands for Contrastive Language-Image Pre-training, which is a pre-training method or model based on contrastive text-image pairs. CLIP is a multimodal model based on contrastive learning. Unlike some contrastive learning methods in computer vision, such as moco and simclr, CLIP's training data is text-image pairs: an image and its corresponding text description. Here, the goal is to learn the matching relationship between text-image pairs through contrastive learning. After training, by directly inputting text containing facial attributes, CLIP performs feature editing on the image features based on the facial attributes, obtaining image features containing text feature information corresponding to the facial attributes.

[0056] Specifically, the fully connected layer acts as a "feature extractor" in the entire convolutional neural network. The fully connected layer can map the learned feature representations to the label space of the samples. In this embodiment, the label space of the samples is the face feature space.

[0057] 105. Generate corresponding facial images based on facial features.

[0058] In this embodiment, generating a corresponding face image based on the face features includes: acquiring at least one random noise, and generating at least one face image based on at least one random noise and the face features through the face generation network.

[0059] Specifically, facial images can be generated directly based on facial features. Alternatively, random noise can be input, and it can be mixed with facial features in different proportions before being input into the face generation network. The purpose of mixing random noise is to achieve multiple outputs from a single sketch input. By inputting a sketch, multiple facial images with similar outlines but differences can be automatically generated via the "More Outputs" button, providing users with more choices. This embodiment uses an example where random noise is mixed into the input face generation network.

[0060] Furthermore, the face generation network consists of multiple generation modules, each consisting of a feature layer and an upsampling layer. The process of generating at least one face image based on the at least one random noise and the face features via the face generation network includes: superimposing the random noise onto the feature layer of each generation module using weight demodulation; inputting a preset fixed input into the feature layer of the first generation module, and adding the face features to the feature layer using adaptive normalization to generate a corresponding face feature image, which is then processed by the corresponding upsampling layer and used as the input to the next generation module; repeating the face feature image generation steps until the last generation module outputs, and using the face feature image output by the last generation module as the face image.

[0061] Specifically, this method uses a Generative Adversarial Network (GAN), which includes two sub-networks, G (Generator) and D (Discriminator). The Generator is a network that generates images. It receives random noise z and generates an image from this noise, denoted as G(z). It consists of 18 modules, each composed of two feature layers and one upsampling layer. The random noise is superimposed onto the feature layers through weight demodulation to generate faces. The Discriminator is a network that determines whether an image is "real." Its input is x, representing an image, and its output D(x) represents the probability that x is a real image. A value of 1 indicates that it is 100% real, while an output of 0 indicates that it is impossible for it to be real. During training, the goal of the Generator is to generate realistic images to deceive the Discriminator D. The goal of the D network is to distinguish the images generated by the G network from real images. Thus, G and D constitute a dynamic "game," with G's ability to forge images continuously improving. When the image generated by the G network can pass the discrimination of the D network, the G network is used as the face generation model.

[0062] In this embodiment, the following steps are taken: First, an input sketch and facial attributes are obtained. The sketch is then input into a preset face generation model. The face generation model calculates token sequences for each partition of the sketch and calculates the global attention of the sketch based on these token sequences. Image features of the sketch are calculated based on the global attention. These image features are mapped to a face feature space, and the mapped image features are then edited to obtain the corresponding facial features of the sketch. Finally, a corresponding facial image is generated based on these facial features. This method achieves high-quality image feature extraction by calculating the global attention of the sketch, reducing the requirements for sketch quality. It can generate high-quality facial images from rough or even incomplete freehand sketches and improves the editability of the generated face through feature editing.

[0063] The face image generation method in the embodiments of the present invention has been described above. The face image generation apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the face image generation device in this invention includes:

[0064] The acquisition module 201 is used to acquire the input sketch and face attributes;

[0065] Attention calculation module 202 is used to input a sketch into a preset face generation model, calculate the token sequence of each partition in the sketch through the face generation model, and calculate the global attention of the sketch based on the token sequence.

[0066] Feature calculation module 203 is used to calculate image features of the sketch based on the global attention;

[0067] The mapping module 204 is used to map the image features to a face feature space, and to perform feature editing on the image features mapped to the face feature space to obtain the face features corresponding to the sketch;

[0068] The face generation module 205 is used to generate a corresponding face image based on the face features.

[0069] In this embodiment of the invention, the face image generation device operates the aforementioned face image generation method. The device acquires an input sketch and face attributes; inputs the sketch into a preset face generation model; calculates the token sequence of each partition in the sketch using the face generation model; calculates the global attention of the sketch based on the token sequence; calculates the image features of the sketch based on the global attention; maps the image features to a face feature space; and performs feature editing on the mapped image features to obtain the corresponding face features of the sketch; finally, it generates the corresponding face image based on the face features. In this method, high-quality image feature extraction is achieved by calculating the global attention of the sketch, reducing the requirements for sketch quality. High-quality facial images can be generated from rough or even incomplete freehand sketches, and the editable range of the generated face is improved through feature editing.

[0070] Please see Figure 3 A second embodiment of the face image generation device in this invention includes:

[0071] The acquisition module 201 is used to acquire the input sketch and face attributes;

[0072] Attention calculation module 202 is used to input a sketch into a preset face generation model, calculate the token sequence of each partition in the sketch through the face generation model, and calculate the global attention of the sketch based on the token sequence.

[0073] Feature calculation module 203 is used to calculate image features of the sketch based on the global attention;

[0074] The mapping module 204 is used to map the image features to a face feature space, and to perform feature editing on the image features mapped to the face feature space to obtain the face features corresponding to the sketch;

[0075] The face generation module 205 is used to generate a corresponding face image based on the face features.

[0076] Specifically, the face generation model includes a feature extraction network, a feature editing network, and a face generation network.

[0077] Specifically, the feature extraction network consists of M stages and N style blocks, where each stage comprises a patch embedding layer and a Transformer Encoder, and M and N are both natural numbers greater than 1; the attention calculation module 202 is specifically used for:

[0078] The sketch is input into a preset face generation model, and the token sequence of the corresponding partition in the sketch is calculated through the patch embedding layer of M stages in the feature extraction network.

[0079] The global attention of the sketch is calculated by the Transformer Encoder in the stage based on the token sequence.

[0080] Specifically, the feature editing network consists of a clip model and fully connected layers; the mapping module 204 is specifically used for:

[0081] The image features are mapped to the face feature space;

[0082] The textual feature information of the facial attributes is extracted using the clip model in the feature editing network.

[0083] The text feature information is mapped to the face feature space through the fully connected layer in the feature editing network, and the corresponding face features of the sketch are obtained based on the image features and text feature information in the face feature space.

[0084] Specifically, the face generation module 205 is used for:

[0085] At least one random noise is acquired, and at least one face image is generated by the face generation network based on at least one of the random noise and the face features.

[0086] Specifically, the face generation network consists of multiple generation modules, and each generation module consists of a feature layer and an upsampling layer;

[0087] The face generation module 205 is further used for:

[0088] The random noise is superimposed onto the feature layer of each generation module through weight demodulation;

[0089] The preset fixed input is input into the feature layer of the first generation module, and the facial features are added to the feature layer through adaptive normalization to generate the corresponding facial feature image. After being processed by the corresponding upsampling layer, it is used as the input of the next generation module.

[0090] Repeat the process of generating facial feature images until the last generation module outputs its image. Use the facial feature image output by the last generation module as the facial image.

[0091] Specifically, the face generation device further includes a model training module 206, which specifically includes:

[0092] The preprocessing unit 2061 is used to acquire sample face images and preprocess the sample face images to obtain sample sketches.

[0093] The input unit 2062 is used to input the sample sketch into a preset neural network model to obtain a model-generated image;

[0094] The loss calculation unit 2063 is used to calculate a preset loss function based on the image generated by the model and the sample face image, obtain the corresponding loss function value, and determine whether the loss function value is less than a preset loss threshold.

[0095] The backpropagation unit 2064 is used to backpropagate the loss function value and update the network parameters of the neural network model if the loss function value is not less than a preset loss threshold, until the obtained loss function value is less than the preset threshold.

[0096] The model generation unit 2065 is used to construct a face generation model based on the network parameters of the neural network model if the loss function value is less than a preset loss threshold.

[0097] Specifically, the preprocessing unit 2061 is used for:

[0098] Acquire sample face images and process them using preset filters to obtain preliminary sketches;

[0099] The preliminary sketch is simplified to obtain a simplified sketch;

[0100] The simplified sketch is augmented with samples to obtain the sample sketch.

[0101] Specifically, the preprocessing unit 2061 is further configured to: extract the edges of the preliminary sketch using the Canny operator;

[0102] Use the fincounters function to identify all connected edges in the given edge;

[0103] Calculate the edge length of each connected edge, and remove connected edges whose edge length is less than a preset length threshold;

[0104] A simplified sketch is generated based on the remaining connected edges after culling.

[0105] This embodiment, based on the previous embodiment, describes in detail the specific functions of each module and the unit composition of some modules. By calculating the global attention of the sketch, it achieves high-quality extraction of image features, reducing the requirements for sketch quality. It can generate high-quality facial images from rough or even incomplete freehand sketches. The feature editing function improves the editable range of the generated face. Combined with random noise, it achieves multiple outputs from a small number of sketches, automatically generating multiple face images with similar but different outlines, providing users with more choices. Furthermore, during model training, by simplifying the sample face images, it automatically generates sketches of various styles for model training. This allows the model to adapt to sketches of different qualities while maintaining high definition and flexible editing capabilities, and it also has good generalization ability.

[0106] above Figure 2 and Figure 3 The face image generation device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The face image generation device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0107] Figure 4 This is a schematic diagram of the structure of a face image generation device 400 provided in an embodiment of the present invention. The face image generation device 400 can vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 410 (e.g., one or more processors) and a memory 420, and one or more storage media 430 (e.g., one or more mass storage devices) for storing application programs 433 or data 432. The memory 420 and storage media 430 can be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the face image generation device 400. Furthermore, the processor 410 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the face image generation device 400 to implement the steps of the aforementioned face image generation method.

[0108] The face image generation device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input / output interfaces 460, and / or one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 4The structure of the face image generation device shown does not constitute a limitation on the face image generation device provided by the present invention. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0109] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the face image generation method.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a human face image, characterized in that, The face image generation method includes: Obtain the input sketch and face attributes; The sketch is input into a preset face generation model. The face generation model calculates the token sequence of each partition in the sketch and calculates the global attention of the sketch based on the token sequence. The face generation model includes a feature extraction network, a feature editing network, and a face generation network. Calculate the image features of the sketch based on the global attention; The image features are mapped to a face feature space, and the image features mapped to the face feature space are edited to obtain the face features corresponding to the sketch. Generate a corresponding facial image based on the facial features; The feature extraction network consists of M stages and N style blocks. Each stage consists of a pattern bedding layer and a Transformer Encoder. M and N are both natural numbers greater than 1. The step of inputting the sketch into a preset face generation model, calculating the token sequence of each partition in the sketch using the face generation model, and calculating the global attention of the sketch based on the token sequence includes: The sketch is input into a preset face generation model, and the token sequence of the corresponding partition in the sketch is calculated through the Patchembedding layer of M stages in the feature extraction network; The global attention of the sketch is calculated by the Transformer Encoder in the stage based on the token sequence.

2. The face image generation method according to claim 1, characterized in that, The feature editing network consists of a clip model and fully connected layers; The step of mapping the image features to a face feature space and then performing feature editing on the image features mapped to the face feature space to obtain the corresponding face features of the sketch includes: The image features are mapped to the face feature space; The textual feature information of the facial attributes is extracted using the clip model in the feature editing network. The text feature information is mapped to the face feature space through the fully connected layer in the feature editing network, and the corresponding face features of the sketch are obtained based on the image features and text feature information in the face feature space.

3. The face image generation method according to claim 1, characterized in that, The step of generating a corresponding facial image based on the facial features includes: At least one random noise is acquired, and at least one face image is generated by the face generation network based on at least one of the random noise and the face features.

4. The face image generation method according to claim 3, characterized in that, The face generation network consists of multiple generation modules, and each generation module consists of a feature layer and an upsampling layer. The step of generating at least one face image based on at least one random noise and the face features through the face generation network includes: The random noise is superimposed onto the feature layer of each generation module through weight demodulation; The preset fixed input is input into the feature layer of the first generation module, and the facial features are added to the feature layer through adaptive normalization to generate the corresponding facial feature image. After being processed by the corresponding upsampling layer, it is used as the input of the next generation module. Repeat the process of generating facial feature images until the last generation module outputs its image. Use the facial feature image output by the last generation module as the facial image.

5. The face image generation method according to any one of claims 1-4, characterized in that, Before acquiring the input sketch and inputting it into the preset face generation model, the process also includes: Acquire sample face images and preprocess the sample face images to obtain sample sketches; The sample sketch is input into a preset neural network model to obtain the model-generated image; The model generates an image and the sample face image, and a preset loss function is calculated to obtain the corresponding loss function value. Determine whether the value of the loss function is less than a preset loss threshold; If not, the loss function value is backpropagated to update the network parameters of the neural network model until the obtained loss function value is less than a preset threshold. If so, a face generation model is constructed based on the network parameters of the neural network model.

6. The face image generation method according to claim 5, characterized in that, The process of acquiring sample face images and preprocessing them to obtain sample sketches includes: Acquire sample face images and process them using preset filters to obtain preliminary sketches; The preliminary sketch is simplified to obtain a simplified sketch; The simplified sketch is augmented with samples to obtain a sample sketch.

7. The face image generation method according to claim 6, characterized in that, The process of simplifying the preliminary sketch to obtain a simplified sketch includes: Use the Canny operator to extract the edges of the preliminary sketch; Use the fincounters function to identify all connected edges in the given edge; Calculate the edge length of each connected edge, and remove connected edges whose edge length is less than a preset length threshold; A simplified sketch is generated based on the remaining connected edges after culling.

8. A face image generation device, characterized in that, The face image generation device includes: The acquisition module is used to acquire the input sketch and face attributes; The attention calculation module is used to input the sketch into a preset face generation model, calculate the token sequence of each partition in the sketch through the face generation model, and calculate the global attention of the sketch based on the token sequence. The face generation model includes a feature extraction network, a feature editing network, and a face generation network. The feature calculation module is used to calculate the image features of the sketch based on the global attention. The mapping module is used to map the image features to a face feature space, and to perform feature editing on the image features mapped to the face feature space to obtain the corresponding face features of the sketch; A face generation module is used to generate a corresponding face image based on the facial features; The feature extraction network consists of M stages and N style blocks. Each stage consists of a pattern bedding layer and a Transformer Encoder. M and N are both natural numbers greater than 1. The attention calculation module is specifically used for: The sketch is input into a preset face generation model, and the token sequence of the corresponding partition in the sketch is calculated through the Patchembedding layer of M stages in the feature extraction network; The global attention of the sketch is calculated by the Transformer Encoder in the stage based on the token sequence.

9. A face image generation device, characterized in that, The face image generation device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the face image generation device to perform the steps of the face image generation method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the face image generation method as described in any one of claims 1-7.

Images