A face editor training, face editing, live broadcast method and related devices

Abstract

An embodiment of the present invention provides a method for training, editing, and live broadcasting a face editor and a related device. The training method of the face editor includes: using original face data as a sample to train a face reconstructor for restoring the face, training a latent vector for the original face data, the latent vector representing the face attributes, searching for a normal vector in the space where the latent vector is located, the normal vector representing the direction of editing each of the face attributes, adjusting the latent vector along the normal vector for each face attribute to generate target face data in the face reconstructor, and adjusting the face reconstructor to a face editor for editing face attributes under the supervision of paired original face data and target face data for each face attribute. By independently training the face editor for a single face attribute, the face editor can maintain better performance when applying a simple structure for supervised learning, the effect of editing face attributes is controllable, and the structure of the face editor can be greatly simplified.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of computer vision, and in particular to a face editor training, face editing, live broadcast method and related devices. Background Art

[0002] In scenarios such as taking photos, recording videos, making short videos, and live streaming, customized tools can be used to edit facial attributes. For example, gender can be enhanced / weakened / changed, aging or rejuvenation can be achieved, eye bags can be increased or decreased, eyes can be enlarged or narrowed, weight can be increased or decreased, makeup can be increased or decreased, and so on. This can enhance business effects such as entertainment, video diversity, and live streaming interaction.

[0003] Because human faces have many attributes, in order to take into account the adjustment of different attributes, the structure of the tools for editing facial attributes is relatively large, and a large number of samples are required during training. Moreover, running tools with such a large structure consumes a lot of resources and takes a long time. They are usually deployed on higher-performance personal computers or the cloud. After users collect image data or video data on the client, they upload it to the personal computer or the cloud to edit the facial attributes, and then transmit it back to the client for display, which causes a certain delay. Summary of the Invention

[0004] The embodiments of the present invention provide a face editor training, face editing, live broadcast method and related devices to solve the problem that the structure of tools for editing facial attributes is relatively large.

[0005] In a first aspect, an embodiment of the present invention provides a method for training a face editor, comprising:

[0006] Using original face data as samples, a face reconstruction tool is trained to restore faces.

[0007] Training a latent vector for the original face data, wherein the latent vector represents a face attribute;

[0008] Finding a normal vector in the space where the latent vector is located, where the normal vector represents a direction for editing each of the facial attributes;

[0009] For each of the facial attributes, adjusting the latent vector along the normal vector to generate target facial data in the face reconstructor;

[0010] For each of the facial attributes, under the supervision of the paired original facial data and the target facial data, the face reconstructor is adjusted to a face editor for editing the facial attribute.

[0011] In a second aspect, an embodiment of the present invention further provides a face editing method, comprising:

[0012] receiving original image data, wherein the original image data includes a human image;

[0013] Selecting a facial attribute for editing the character image from a plurality of facial attributes as a target attribute, each of the facial attributes being configured with a face editor for editing the facial attribute;

[0014] loading a face editor for editing the target attributes;

[0015] The original image data is input into the face editor for editing to edit the target attributes of the character image and obtain target image data.

[0016] In a third aspect, an embodiment of the present invention further provides a live broadcast method, comprising:

[0017] Collecting video data, wherein multiple frames of original image data in the video data contain a host user;

[0018] Selecting a facial attribute to be edited by the anchor user from a plurality of facial attributes as a target attribute, each of the facial attributes being configured with a face editor for editing the facial attribute;

[0019] loading a face editor for editing the target attributes;

[0020] Inputting the original image data into the face editor for editing to edit the target attributes of the anchor user and obtain target image data;

[0021] Preview the video data containing the target image data, and stream the video data containing the target image data to the live broadcast room opened by the anchor user for publishing.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer device, comprising:

[0023] one or more processors;

[0024] a memory for storing one or more programs,

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the face editor as described in the first aspect, the face editing as described in the second aspect, or the live broadcast method as described in the third aspect.

[0026] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the training method of the face editor as described in the first aspect, the face editing as described in the second aspect, or the live broadcast method as described in the third aspect.

[0027] In this embodiment, a face reconstructor for restoring faces is trained using original facial data as samples. Latent vectors are trained on the original facial data. The latent vectors represent facial attributes. A normal vector is searched for in the space of the latent vectors. The normal vector represents the direction for editing each facial attribute. For each facial attribute, the latent vector is adjusted along the normal vector to generate target facial data in the face reconstructor. For each facial attribute, under the supervision of paired original and target facial data, the face reconstructor is adjusted to a face editor for editing facial attributes. By independently training the face editor for a single facial attribute, the face editor can maintain excellent performance while employing a simple structure for supervised learning. The effect of editing facial attributes is controllable, significantly simplifying the face editor structure and reducing the number of samples used in training. Furthermore, the simpler face editor consumes fewer resources and takes less time to run. It can be deployed on higher-performance personal computers or the cloud, or on lower-performance mobile devices. Editing facial attributes on a mobile device eliminates the need to upload data to a personal computer or the cloud, resulting in less time and a faster response.

[0028] In addition, the training process of this embodiment is simple, and new facial attributes can be quickly iterated as new special effects, which helps to quickly launch new special effects and avoids a lot of lengthy pre-processing and post-processing, providing a low-cost, high-efficiency industrialized process for new special effects in business scenarios such as live broadcasts and short videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of the architecture of existing tools for editing facial attributes;

[0030] Figure 2 A flowchart of a face editor training method provided in Example 1 of the present invention;

[0031] Figure 3A A schematic diagram of the architecture of a face reconstructor provided in the first embodiment of the present invention;

[0032] Figure 3B A schematic diagram of the architecture of a decoder provided in the first embodiment of the present invention;

[0033] Figure 3C A schematic diagram of the architecture of a discriminator provided in the first embodiment of the present invention;

[0034] Figure 4 A schematic diagram of a cyclic operation provided in Example 1 of the present invention;

[0035] Figure 5 This is an example diagram of editing facial attributes provided by the first embodiment of the present invention;

[0036] Figure 6 This is an example diagram of a supervised learning sample provided in the first embodiment of the present invention;

[0037] Figure 7 A schematic diagram of the architecture of a face editor provided in the first embodiment of the present invention;

[0038] Figure 8 This is a flowchart of a face editor training method provided in the second embodiment of the present invention;

[0039] Figure 9 This is an example diagram of facial attribute coupling and decoupling provided by the second embodiment of the present invention;

[0040] Figure 10 is a schematic diagram of a dissociated normal vector provided by Example 2 of the present invention;

[0041] Figure 11 This is a flowchart of face editing provided by the third embodiment of the present invention;

[0042] Figure 12 This is a flow chart of a live broadcast method provided by the fourth embodiment of the present invention;

[0043] Figure 13 A schematic structural diagram of a face editor training device provided in a fifth embodiment of the present invention;

[0044] Figure 14 A schematic structural diagram of a face editing device provided in a sixth embodiment of the present invention;

[0045] Figure 15 A schematic structural diagram of a live broadcast device provided in Embodiment 7 of the present invention;

[0046] Figure 16 A schematic diagram of the structure of a computer device provided in Example 8 of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0048] like Figure 1As shown, tools for editing facial attributes generally include a decoder and an encoder. In addition to the decoder and encoder, an attribute editing module needs to be additionally trained to provide a unified editing method for numerous facial attributes. The image data is input into the encoder and encoded into a vector. The attribute editing module edits the vector and adds features of the facial attributes that the user intends to edit. After the editing is completed, the vector is input into the decoder and decoded to obtain the effect image the user wants, such as an aging effect image, a male effect image, a beard effect image, an effect image with glasses, and so on.

[0049] In order to take into account different facial attributes, the structures of the decoder and encoder are relatively complex, and the processing method of the attribute editing module is also relatively complex.

[0050] Example 1

[0051] Figure 2 This is a flowchart of a face editor training method provided in the first embodiment of the present invention. This embodiment is applicable to the case of training a face editor for each facial attribute. The method can be performed by a face editor training device. The face editor training device can be implemented by software and / or hardware and can be configured in a computer device, such as a server, workstation, personal computer, etc., and specifically includes the following steps:

[0052] Step 201: Use original face data as samples to train a face reconstructor for restoring faces.

[0053] To facilitate the collection of a sufficient number of data sets, image data representing faces can be collected from open source databases (such as FFHQ (Flickr Faces HighQuality)) and / or open source projects as original face data. Of course, to improve the accuracy of face reconstructors and face editors in business scenarios, image data representing faces can also be collected through channels of business scenarios as original face data. This embodiment does not impose any restrictions on this.

[0054] Taking live broadcast as an example of a business scenario, image data containing the face of the anchor user when the anchor user hosts the live broadcast program can be collected as the original facial data.

[0055] In addition, considering that the users of the face editor in the business scenario are regional, the original face data in the corresponding region can be collected.

[0056] For example, assuming that the face editor is mainly for Asian users, 30,000 original face data can be collected from the open source database FFHQ, 30,000 original face data can be collected from the open source project - Asian Generator, and 40,000 original face data can be collected from anchor users, thus forming a data set of 100,000.

[0057] In this embodiment, the original face data can be used as training samples to train a face reconstructor, which is used to restore the face. That is, given the face data, the face reconstructor can extract features and use the features to regenerate the face data.

[0058] In one embodiment of the present invention, the face reconstructor includes an encoder and a decoder. In this embodiment, step 201 includes the following steps:

[0059] Step 2011: In the encoder, the original face data is encoded into a vector as a face vector.

[0060] In this embodiment, the original facial data may be input into an encoder, which encodes the original facial data and extracts a vector of the original facial data as a feature of the face. The vector is recorded as a face vector.

[0061] In an encoder, such as Figure 3A As shown, the encoder includes n (n is a positive integer) convolutional layers 301 and one fully connected layer (FC) 302, wherein the n convolutional layers 301 are connected in sequence. In the order of forward propagation, the dimension of the output vector of the convolutional layer 301 that is sorted first is greater than the dimension of the output vector of the convolutional layer 301 that is sorted later.

[0062] Then, in the encoder, n convolutional layers 301 are called in sequence according to the order to perform convolution processing on the original face data to obtain a candidate vector. That is, the original face data is input into the first convolutional layer 301 for convolution processing, and the vector of the original face data is output. For the second to n convolutional layers 301, the vector output by the previous convolutional layer is received, and the feature is convolved and a new vector is output. The vector output by the nth convolutional layer is recorded as a candidate vector.

[0063] The candidate vector is output to the fully connected layer 302, which then calls the fully connected layer 302 to perform a fully connected operation on the candidate vector to obtain a 2n-layer face vector 303.

[0064] Exemplarily, let n = 9, the dimension of the vector output by the first convolutional layer is 1024×1024×512, the dimension of the vector output by the second convolutional layer is 512×512×512, the dimension of the vector output by the third convolutional layer is 256×256×512, the dimension of the vector output by the fourth convolutional layer is 128×128×512, the dimension of the vector output by the fifth convolutional layer is 64×64×512, the dimension of the vector output by the sixth convolutional layer is 32×32×512, the dimension of the vector output by the seventh convolutional layer is 16×16×512, the dimension of the vector output by the eighth convolutional layer is 8×8×512, the dimension of the vector output by the ninth convolutional layer is 4×4×512, and the dimension of the vector output by the fully connected layer is 18×512, so that the dimension of the face vector is 18×512.

[0065] Step 2012: In the decoder, the face vector is decoded into face data as reconstructed face data.

[0066] In this embodiment, the face vector can be input into the decoder, and the decoder decodes the original face data to generate new image data representing the face as face data, which is recorded as reconstructed face data.

[0067] In a decoder, such as Figure 3A As shown, the decoder is the synthesis network in the style adversarial generation network StyleGAN2, and the synthesis network includes n blocks Block 304, and the n blocks Block 304 are connected in sequence.

[0068] like Figure 3B As shown, each block Block 304 can be divided into an upper half and a lower half, and each half can be divided into a convolutional layer Conv (the convolution kernel can be 3×3) and a style converter AdaIN (AdaIN is a variant of IN (Instance Norm, which normalizes the height H and width W of image pixels and is used for style transfer)).

[0069] Since the convolutional layers Conv of the two parts are different, the upper part is called Conv0 and the lower part is called Conv1. The first block does not have Conv0.

[0070] like Figure 3AAs shown, in the decoder, n blocks 304 are called in sequence to process the 2n-layer face vectors 303 to obtain reconstructed face data, wherein every 2 layers of face vectors 303 are the input of each block 304, and in particular, each face vector 303 is the input of each style converter AdaIN in turn.

[0071] Specifically, if Figure 3B As shown, for the first block Block 304, noise Noise (identifier B) is added to the representation variable Const, and in the style converter AdaIN, the face variable (identifier A) of the 0th layer is used to perform style transfer on the face data, and in the convolution layer Conv, the face data is convolved and noise Noise (identifier B) is added to the face data, and in the style converter AdaIN, the face variable (identifier A) of the 1st layer is used to perform style transfer on the face data, and the face data is output to the next block Block 304.

[0072] For the i-th (i>1) block Block 304, in the convolution layer Conv, the face data (Upsample) output by the i-1-th block Block304 is convolved, and noise Noise (identifier B) is added to the face data. In the style converter AdaIN, the face variable (identifier A) of the 2i-2 layer is used to perform style transfer on the face data. In the convolution layer Conv, the face data is convolved, and noise Noise (identifier B) is added to the face data. In the style converter AdaIN, the face variable (identifier A) of the 2i-1 layer is used to perform style transfer on the face data, wherein the number of layers for face changing is set starting from the 0th layer.

[0073] If the i-th block Block 304 is not the last block Block 304 , the facial data is output to the next block Block 304 ; if the i-th block Block is the last block Block 304 , the facial data is output as reconstructed facial data.

[0074] like Figure 3A As shown in the figure, the structure of the decoder is opposite to that of the encoder. If in the encoder, in the order of forward propagation, the dimension of the output vector of the convolutional layer that is sorted first is larger than the dimension of the output vector of the convolutional layer that is sorted later, then in the decoder, in the order of forward propagation, the dimension of the output vector of the block Block304 that is sorted first is smaller than the dimension of the output vector of the block Block304 that is sorted later.

[0075] For example, let n=9, the first block inputs the face vectors of the 0th and 1st layers, and the dimension of the output vector is 4×4×512, the second block inputs the face vectors of the 2nd and 3rd layers, and the dimension of the output vector is 8×8×512, the third block inputs the face vectors of the 4th and 5th layers, and the dimension of the output vector is 16×16×512, the fourth block inputs the face vectors of the 6th and 7th layers, and the dimension of the output vector is 32×32×512, the fifth block inputs the face vectors of the 8th and 9th layers, and the dimension of the output vector is 16×16×512. The dimension of the vector is 64×64×512. The 6th block inputs the face vectors of the 10th and 11th layers, and the dimension of the output vector is 128×128×512. The 7th block inputs the face vectors of the 12th and 13th layers, and the dimension of the output vector is 256×256×512. The 8th block inputs the face vectors of the 14th and 15th layers, and the dimension of the output vector is 512×512×512. The 9th block inputs the face vectors of the 16th and 17th layers, and the dimension of the output vector is 1024×1024×512.

[0076] Step 2013: In the universal discriminator, a loss value is calculated for the original face data and the reconstructed face data as a reconstruction loss value.

[0077] When training a face reconstructor, a discriminator can be set, denoted as a universal discriminator. The face data (original face data, reconstructed face data) is input into the universal discriminator, and universal face features are extracted from the original face data. Based on the universal face features, it is determined whether the face data is real face data (original face data) or false face data (reconstructed face data).

[0078] To improve the efficiency of training face reconstructor, such as Figure 3C As shown, the universal discriminator includes n convolutional layers 305, and the n convolutional layers of the encoder have the same structure as the n convolutional layers of the discriminator, wherein the n convolutional layers are connected in sequence, and in the order of forward propagation, the dimension of the output vector of the convolutional layer that is sorted first is greater than the dimension of the output vector of the convolutional layer that is sorted later.

[0079] Exemplarily, in the universal discriminator, a first universal facial feature is extracted from the original facial data, and a second universal facial feature is extracted from the reconstructed facial data. Based on the first universal facial feature and the second universal facial feature, L2 (Euclidean) distance, perceptual loss (Perceptual Losses) and other loss values ​​are calculated as reconstruction loss values.

[0080] Step 2014: Determine whether the reconstruction loss value converges; if so, execute step 2015; if not, execute step 2016.

[0081] Step 2015: Determine whether the face reconstructor training is completed.

[0082] Step 2016: Update the decoder and encoder, and return to step 2011.

[0083] In this embodiment, conditions indicating convergence can be set in advance for the reconstruction loss value, for example, the reconstruction loss value is less than a first general threshold, the difference between adjacent reconstruction loss values ​​is recorded as the change amplitude, multiple consecutive change amplitudes are less than a second general threshold, the number of iterations exceeds a third general threshold, and so on.

[0084] In each iteration, it can be determined whether the current reconstruction loss value meets the condition.

[0085] If the condition is met, the reconstruction loss value converges. At this time, it is confirmed that the face reconstructor training is completed, and the structure of the encoder and decoder and the parameters of the encoder and decoder in the face reconstructor are stored.

[0086] If this condition is not met, the encoder and decoder in the face reconstructor are back-propagated, and the parameters of the decoder and encoder are updated by manually setting the learning rate optimization method represented by the stochastic gradient descent (SGD) algorithm, or adaptively setting the learning rate represented by the adaptive moment estimation (Adam) algorithm, and then entering the next iteration.

[0087] Step 202: Train latent vectors on the original face data.

[0088] The face reconstructor is used to restore the human face. The face vector output by its encoder does not necessarily represent the facial attributes. To edit the facial attributes, the original face data can be latent through methods such as iterative optimization and training of the face reconstructor. The latent vector latent represents facial attributes, such as facial features, face shape, skin color, etc.

[0089] In one embodiment of the present invention, a latent vector representing facial attributes of original facial data can be obtained by combining methods such as an iterative optimization method and a trained face reconstructor. This can simultaneously take into account the advantages of ensuring the effect of reconstructing the latent vector based on the iterative optimization method and reducing the processing time based on the trained face reconstructor. In this embodiment, step 202 may include the following steps:

[0090] Step 2021: Execute loop operation.

[0091] In this embodiment, if Figure 4 As shown, a loop operation is set, which refers to S1→S2→S3→S1→S2→S3→S1..., where the arrows indicate the order of execution, that is, in one loop operation, S1 is executed first, then S2, and then S3, and after executing S3, the next loop operation can be entered.

[0092] S1. The encoder in the face reconstructor encodes the original face data into a face vector.

[0093] S2. Map the face vector to a latent vector according to the mapping parameters (such as a vector matrix).

[0094] S3. The decoder in the face reconstructor decodes the latent vector into new original face data.

[0095] When executing a loop operation, it is not necessary to execute the loop operation completely once. The loop operation can be stopped when S1 or S2 is completed.

[0096] Step 2022: Generate a loss value based on the loop operation as the attribute loss value.

[0097] In this embodiment, at least one loop operation can be performed so that new original face data can be reconstructed using latent vector decoding. Subsequently, the new original face data can be aligned with the real original face data in each operation to calculate the difference between the two and obtain a loss value, which is recorded as the attribute loss value.

[0098] In one example, the attribute loss value includes the following three parts:

[0099] In the first part, a first distance between the original face data and the new original face data is calculated.

[0100] In the second part, the third distance representing the perceptual loss is calculated between the original face data and the new original face data.

[0101] In a specific implementation, a general feature extraction network can be set, such as the VGG (Visual Geometry Group) network in CNN (Convolutional Neural Networks) to extract higher-dimensional features.

[0102] On the one hand, the original face data is input into the universal feature extraction network to extract features and obtain the first universal image feature. On the other hand, the new original face data is input into the universal feature extraction network to extract features and obtain the second universal image feature, thereby calculating the second distance between the first universal image feature and the second universal image feature.

[0103] In the third part, the third distance between the latent vector corresponding to the original face data (i.e., the latent vector after performing S1 and S2 on the original face data) and the latent vector corresponding to the new original face data (i.e., the latent vector after performing S1 and S2 on the new original face data) is calculated.

[0104] The first distance, second distance, and third distance are linearly fused into the attribute loss value. That is, the first distance, second distance, and third distance are assigned corresponding weights and then summed. Generally, the weight of the first distance is greater than the weight of the second distance and the weight of the third distance.

[0105] For example, the attribute loss value is expressed as follows:

[0106] LOSS attribute =||xG(z)||2+λ vgg ||F(x)-F(G(z))||2+λ dom ||zE(G(z))||2

[0107] Among them, LOSS attribute is the attribute loss value, x is the original face data, z is the latent vector, E is the encoder, G is the decoder, F is the general feature extraction network (such as VGG network), λ vgg is the weight, a hyperparameter, λ dom is the weight and is a hyperparameter.

[0108] ||xG(z)||2 is the first distance, ||F(x)-F(G(z))||2 is the second distance, and ||zE(G(z))||2 is the third distance.

[0109] Step 2023: Determine whether the attribute loss value converges; if so, execute step 2024; if not, execute step 2025.

[0110] Step 2024: Determine whether latent vector training is complete.

[0111] Step 2025: Update the mapping parameters and return to execute step 2021.

[0112] In this embodiment, with the goal of minimizing the attribute loss value, conditions representing convergence can be set in advance for the attribute loss value, for example, the attribute loss value is less than the first attribute threshold, the difference between adjacent attribute loss values ​​is recorded as the change amplitude, multiple consecutive change amplitudes are less than the second attribute threshold, the number of iterations exceeds the third attribute threshold, and so on.

[0113] In each iteration, it can be determined whether the current attribute loss value meets the condition.

[0114] If this condition is met, the attribute loss value converges. At this time, the impression training is confirmed to be completed and the latent vector is stored.

[0115] At this time, the latent vector can be expressed as:

[0116]

[0117] Among them, arg min represents the attribute loss value LOSS attribute The variable value (i.e., latent vector) when it takes the minimum value.

[0118] If this condition is not met, the mapping parameters are updated by manually setting the learning rate through an optimization method represented by the stochastic gradient descent algorithm, or by adaptively setting the learning rate represented by the adaptive moment estimation algorithm to enter the next iteration.

[0119] Step 203: Find the normal vector in the space where the latent vector is located.

[0120] In the vector space where the latent vector is located (i.e., latent space), for each facial attribute, the method of operating the latent vector can be used to find the editable facial attribute vector in the vector space, which is denoted as the normal vector. That is, each normal vector represents the direction of editing each facial attribute.

[0121] In one embodiment of the present invention, step 203 may include the following steps:

[0122] Step 2031: Randomly input the latent vector into the decoder in the face reconstructor for decoding to obtain multiple frames of sample face data.

[0123] In this embodiment, the decoder Decoder in the trained face reconstructor is used to randomly use latent vectors to synthesize multiple frames of sample face data, such as 500,000 frames of sample face data. At this time, the relationship between the sample face data image and the latent vector latent is recorded (latent, image).

[0124] Step 2032: For each facial attribute, perform binary classification on the facial data to obtain the facial attribute and confidence level of the facial data.

[0125] In this embodiment, a binary classifier may be pre-trained for each facial attribute, such as SVM (Support Vector Machine), LR (Logistic Regression), and the like.

[0126] For multiple frames of sample face data, the binary classifier can be used to annotate the sample face data with a pseudo label (whether it has the facial attributes) and the corresponding confidence score. At this time, the relationship between the latent vector latent, pseudo label pseudo label and confidence score confidence score is recorded (latent, pseudo label, confidence score).

[0127] Step 2033: Classify part of the sample face data into positive samples and classify part of the sample face data into negative samples.

[0128] The sample face data is sorted according to the confidence level, so that some sample face data are selected as positive samples and some sample face data are classified as negative samples. The confidence level of the positive sample is higher than that of the negative sample. At this time, the relationship between the latent vector and the pseudo label (latent, pseudo label) can be recorded.

[0129] Generally, in order to facilitate the generation of hyperplanes and calculation of normal vectors, the number of positive samples is equal to the number of negative samples.

[0130] For example, 10,000 sample face data with the highest confidence level can be selected as positive samples, and 10,000 sample face data with the lowest confidence level can be selected as negative samples.

[0131] Step 2034: Determine a hyperplane in the space where the latent vector is located.

[0132] A hyperplane is found in the vector space where the latent vector is located. The hyperplane divides the vector space into two subspaces. The subspace on one side of the hyperplane includes the latent vectors of positive samples, and the subspace on the other side of the hyperplane includes the latent vectors of negative samples.

[0133] Step 2035: Set the normal vector of the hyperplane to edit the normal vector of each facial attribute.

[0134] For each facial attribute, the normal vector of the hyperplane is calculated and recorded as the normal vector of the corresponding facial attribute, which represents the direction of facial attribute editing.

[0135] Step 204: For each facial attribute, adjust the latent vector along the normal vector to generate target facial data in the face reconstructor.

[0136] In this embodiment, based on the original facial attributes, the latent vector can be adjusted along the normal vector according to business needs, and the adjusted latent vector can be input into the face reconstructor. The face reconstructor uses the adjusted latent vector to generate new facial data, which is recorded as target facial data. The target facial data is adjusted relative to the original facial data under the facial attributes represented by the normal vector.

[0137] like Figure 5 As shown, for the same frame of original face data, multiple frames of target face data can be generated, among which the first row of target face data editing represents the facial attribute of age, which becomes older to the left and younger to the right. The second row of target face data editing represents the facial attribute of gender, which becomes a man to the left and a woman to the right. The third row of target face data editing represents the facial attribute of mood, which becomes angry to the left and smiling to the right. The fourth row of target face data editing represents the facial attribute of glasses, which becomes not wearing glasses to the left and wearing glasses to the right. The fifth row of target face data editing represents the facial attribute of appearance, which becomes dressing to the left and makeup to the right.

[0138] In a specific implementation, the editing strength can be determined, and the editing strength is used to adjust the strength of adjusting the latent vector along the normal vector. Generally, the editing strength can be set by technical personnel based on business needs, editing effects and other factors. For example, if the business requirement is 20 years of aging, technical personnel can screen the editing strength that achieves 20 years of aging under the dimension of facial attributes representing aging.

[0139] For each facial attribute, the normal vector corresponding to the facial attribute is adjusted using the editing strength. If the adjustment is completed, the normal vector is added to the latent vector to obtain the edited latent vector. That is, the product of the editing strength and the normal vector is added to the original latent vector to obtain the edited latent vector, which is expressed as follows:

[0140] z edit =z+αn

[0141] Among them, z is the original latent vector, z edit is the new hidden vector, α is the editing strength, α>0 is the positive direction, α<0 is the negative direction, and n is the normal vector.

[0142] The adjusted latent vector is input into the decoder in the face reconstructor for decoding to obtain the target face data, which is expressed as follows:

[0143] G(z edit )=G(z+αn)

[0144] Among them, G is a decoder.

[0145] Step 205: For each facial attribute, under the supervision of the paired original facial data and the target facial data, adjust the face reconstructor to a face editor for editing facial attributes.

[0146] For each facial attribute, paired original facial data and target facial data are selected. Pairing means that the target facial data is reconstructed by adjusting the latent vector of the original facial data along the normal vector of the facial attribute.

[0147] For example, Figure 6 As shown in the figure, multiple pairs of original face data (on the left) and target face data (on the right) are selected for the facial attributes of gender (female to male), gender (male to female), age (older), and age (younger) in turn as samples for supervised learning.

[0148] The face reconstructor is supervised by using the paired original face data and the target face data, thereby being adjusted to a face editor for editing the attributes of the face.

[0149] like Figure 7 As shown in the figure, each facial attribute corresponds to a face editor trained. Assuming there are m facial attributes, m face editors can be trained accordingly. The face editor includes a decoder and an encoder. In addition to the decoder and encoder, it does not rely on the attribute editing module, saving the attribute editing operation.

[0150] The image data is input into the encoder of the facial attribute to be edited and encoded into a vector. The vector is input into the decoder corresponding to the encoder and decoded to obtain the effect image desired by the user, such as the effect image of aging, the effect image of becoming male, the effect image of growing a beard, the effect image of wearing glasses, etc.

[0151] In one embodiment of the present invention, step 205 may include the following steps:

[0152] Step 2051: For each facial attribute, the original facial data is input into a face reconstructor to generate facial data as reference facial data.

[0153] For each facial attribute, the original facial data can be input into the face reconstructor respectively, and the face reconstructor regenerates facial data, which is recorded as reference facial data. At this time, the reference facial data is the predicted facial data after editing the facial attributes.

[0154] In a specific implementation, the face reconstructor includes an encoder and a decoder. The original face data is input into the encoder, the original face data is encoded into a vector as a face vector, the face vector is input into the decoder, and the face vector is decoded into face data as reference face data.

[0155] Step 2052: Input the reference face data of multiple resolutions and the target face data of multiple resolutions into multiple personality discriminators respectively to extract the first personality face feature of the reference face data, extract the second personality face feature of the target face data, and output a true or false discrimination result.

[0156] In this embodiment, a face editor can be trained based on a lightweight Pix2PixHD model. To distinguish between real target face data and synthesized reference face data at high resolution, the discriminator is generally required to have a large receptive field, which relies on a deeper network or a larger convolution kernel. Both options will increase the network capacity, making the network more prone to overfitting problems, and the storage space required for training will also increase.

[0157] Therefore, in the Pix2PixHD model, multiple discriminators can be set for multiple resolutions. Multiple discriminators have the same network structure and can process face data of different scales. That is, one discriminator can handle the identification of face data of one resolution. For the sake of distinction, it is recorded as a personality discriminator.

[0158] For example, three personality discriminators may be set, each responsible for face data with resolutions of 256x256, 128x128, and 64x64.

[0159] Although the network structure of the personality discriminator is the same, the coarsest-scale personality discriminator has the largest receptive field, which contains more global image perspective information and can guide the face reconstructor to generate globally consistent face data. The finest-scale personality discriminator encourages the face reconstructor to generate finer details, which also makes it easier to train the face reconstructor from coarse to fine. Therefore, expanding the low-resolution face reconstructor to a high-resolution face reconstructor only requires adding a fine-scale discriminator, rather than retraining from scratch.

[0160] For reference facial data and target facial data, both can be expanded into reference facial data of multiple resolutions and target facial data of multiple resolutions through operations such as scaling. The reference facial data and target facial data at the same resolution are input into the personality discriminator corresponding to the resolution. The personality discriminator extracts the first personality facial feature of the reference facial data and the second personality facial feature of the target facial data, and outputs a discrimination result of whether the reference facial data is true or false based on the first personality facial feature, and outputs a discrimination result of whether the target facial data is true or false based on the second personality facial feature.

[0161] Step 2053: Calculate the loss value of the face reconstructor editing the face attributes based on the discrimination result as the adversarial loss value.

[0162] For the personality discriminator, when discriminating the real target face data, it is hoped that its discrimination result is as close to true (1) as possible. When discriminating the synthesized reference face data, it is hoped that its discrimination result is as close to false (0) as possible. For the face reconstructor, it is hoped that through training, the synthesized reference face data that is the same as the real target face data can be synthesized. Therefore, it is hoped that its discrimination result is as close to true (1) as possible.

[0163] For example, the adversarial loss value can be calculated using a binary cross entropy function, which is expressed as follows:

[0164] L GAN (G,D k )=E[logD k (y)]+E[log(1-D k (G(x)))]

[0165] Among them, L GAN To combat the loss value, G is the face reconstructor, D k is the k-th personality discriminator, E is the operation of calculating the mean, x is the original face data, y is the target face data, and G(x) is the reference face data.

[0166] In the Pix2PixHD model, the sum of the adversarial loss values ​​corresponding to each personality discriminator can be calculated as the final adversarial loss value.

[0167] The lightweight Pix2PixHD model uses OpenVINO's inference engine to process a frame of face data in about 15ms in a single-threaded environment (processor is i7-7700, frequency is 3.60GHz). It can be deployed in three environments: remote, personal computer, and mobile terminal.

[0168] Step 2054: Calculate the feature matching loss value based on the first facial feature and the second facial feature.

[0169] The adversarial loss value of the face reconstructor is based on the output of the personality discriminator, that is, the cross entropy output by the personality discriminator. Since the face reconstructor is given little information and the space of face data is large, this may lead to unstable training of the face reconstructor.

[0170] To make training more stable, we can use the Feature Matching method to calculate the loss value, denoted as Feature Matching LOSS. Feature Matching requires that the first personality facial features extracted from the reference face data generated by the face reconstructor, when passing through the personality discriminator, are as close as possible to (match) the second personality facial features extracted from the target face data when passing through the personality discriminator.

[0171] For example, the adversarial loss value is expressed as follows:

[0172]

[0173] Among them, L FM is the feature matching loss value, G is the face reconstructor, and D k is the kth personality discriminator, E is the operation of calculating the mean, x is the original face data, y is the target face data, G(x) is the reference face data, is the personality discriminator D k The extracted features of the i-th layer (the first facial feature or the second facial feature), N i is the number of features in the i-th layer.

[0174] In the Pix2PixHD model, the sum of the feature matching loss values ​​corresponding to each personality discriminator can be calculated as the final feature matching loss value.

[0175] Step 2055: Calculate the perception loss value based on the target face data and the reference face data.

[0176] In a specific implementation, a personalized feature extraction network can be set, such as the VGG network in

[15] , to extract features of higher dimensions.

[0177] On the one hand, the target face data is input into the personality feature extraction network to extract features and obtain the first personality image feature. On the other hand, the reference face data is input into the personality feature extraction network to extract features and obtain the second personality image feature. The distance between the first personality image feature and the second personality image feature is calculated as the perceptual loss value (Perceptual Losses), which is expressed as follows:

[0178] L VGG (G,F)=‖F(y)-F(G(x))‖1

[0179] Among them, L VGG is the perceptual loss value, G is the face reconstructor, F is the personality feature extraction network, x is the original face data, y is the target face data, and G(x) is the reference face data.

[0180] Step 2056: Linearly fuse the adversarial loss value, feature matching loss value, and perceptual loss value into a total loss value.

[0181] In this embodiment, the anti-loss value, the feature matching loss value and the perception loss value can be linearly fused to obtain a loss value, which is recorded as the total loss value.

[0182] That is, the adversarial loss value, feature matching loss value, and perceptual loss value are configured with corresponding weights and then summed. Generally, the weight of the adversarial loss value is greater than the weight of the feature matching loss value and the weight of the perceptual loss value.

[0183] For example, the total loss value is expressed as follows:

[0184]

[0185] Among them, LOSS total is the total loss value, L GAN To combat the loss value, L FM is the feature matching loss value, L VGG is the perceptual loss value, λ fm is the weight, a hyperparameter, λ vgg is the weight and is a hyperparameter.

[0186] Step 2057: Under the constraint of maximizing the adversarial loss value, determine whether the total loss value converges; if so, execute step 2058; if not, execute step 2059.

[0187] Step 2058: Determine whether the face reconstructor has been adjusted to a face editor for editing facial attributes.

[0188] Step 2059: Update the decoder and encoder, and return to execute step 2051.

[0189] In this embodiment, the face reconstructor is iteratively trained with maximizing the adversarial loss value and minimizing the total loss value as the training goals. During each iterative training process, it can be determined whether the adversarial loss value and the total loss value meet the above goals.

[0190] If satisfied, it can be confirmed that the face reconstructor has been adjusted to a face editor for editing facial attributes, the mapping relationship between facial attributes and the face editor is recorded, and the structure of the encoder and decoder in the face editor and the parameters of the encoder and decoder are stored.

[0191] If it is not satisfied, the encoder and decoder in the face reconstructor are back-propagated, and the parameters of the decoder and encoder are updated through an optimization method of manually setting the learning rate represented by the stochastic gradient descent algorithm, or an adaptive setting of the learning rate represented by the adaptive moment estimation, to enter the next iteration.

[0192] In this embodiment, a face reconstructor for restoring faces is trained using original facial data as samples. Latent vectors are trained on the original facial data. The latent vectors represent facial attributes. A normal vector is searched in the space of the latent vectors. The normal vector indicates the direction for editing each facial attribute. For each facial attribute, the latent vector is adjusted along the normal vector to generate target facial data in the face reconstructor. For each facial attribute, under the supervision of paired original and target facial data, the face reconstructor is adjusted to a face editor for editing facial attributes. By independently training the face editor for a single facial attribute, the face editor can maintain excellent performance while using a simple structure for supervised learning. The effect of editing facial attributes is controllable, significantly simplifying the face editor structure and reducing the number of samples used in training. Furthermore, the simpler face editor consumes fewer resources and takes less time to run. It can be deployed on higher-performance personal computers or the cloud, or on lower-performance mobile devices. Editing facial attributes on a mobile device eliminates the need to upload data to a personal computer or the cloud, resulting in less time and a faster response.

[0193] In addition, the training process of this embodiment is simple, and new facial attributes can be quickly iterated as new special effects, which helps to quickly launch new special effects and avoids a lot of lengthy pre-processing and post-processing, providing a low-cost, high-efficiency industrialized process for new special effects in business scenarios such as live broadcasts and short videos.

[0194] Example 2

[0195] Figure 8This is a flowchart of a face editor training method provided in Example 2 of the present invention. This embodiment is based on the previous embodiment and further adds an operation of dissociating normal vectors. The method specifically includes the following steps:

[0196] Step 801: Use original face data as samples to train a face reconstructor for restoring faces.

[0197] Step 802: Train latent vectors on the original face data.

[0198] Among them, the latent vector represents the attributes of the face.

[0199] Step 803: Find the normal vector in the space where the latent vector is located.

[0200] Among them, the normal vector represents the direction of editing each facial attribute.

[0201] Step 804: For the current normal vector, set other normal vectors as reference vectors.

[0202] Step 805: Separate the reference vector from the current normal vector to use as the separated normal vector.

[0203] In practical applications, although the face vector generated by the face reconstructor for face data can dissociate the coupling state of face attributes in the normal vector to a certain extent, the dissociation of the coupling state depends on the original face data used as the sample. When the amount of sample data is large, the face attributes are difficult to separate, which is especially obvious for video data. That is to say, in the original face data, if two or more face attributes are coupled, the normal vectors corresponding to the face attributes are also coupled.

[0204] For example, Figure 9 As shown in the figure, the facial attribute representing age is edited for a certain original facial data (the facial data selected by the box). Going left means getting older, and going right means getting younger. In the first row of edited facial data, since the proportion of elderly people wearing glasses in the samples for training the decoder is higher than that of young people wearing glasses, the facial attribute representing age is coupled with the facial attribute representing wearing glasses. When the normal vector of the facial attribute representing age is used to adjust the latent vector, the face will become older and wear glasses at the same time.

[0205] Therefore, in this embodiment, coupled facial attributes can be found, and other irrelevant facial attributes can be dissociated from their normal vectors so that the normal vector edits a single facial attribute, ensuring the performance of the face editor, making the editing effect of the facial attributes controllable, and accurately editing the facial attributes required by the user.

[0206] In the specific implementation, for the normal vector that currently has coupled facial attributes, other facial attributes can be queried as reference vectors, where the facial attributes edited by the current normal vector include the facial attributes edited by the reference vector. That is, in addition to editing its own facial attributes, the current normal vector can also edit the facial attributes corresponding to the reference vector.

[0207] Generally, the reference vector is the smallest unit normal vector. The reference vector is used to edit a single facial attribute, and there is no case of editing other coupled facial attributes.

[0208] Extract the reference vector from the current vector to obtain the extracted normal vector.

[0209] In a specific implementation, the transposed matrix of the current normal vector is calculated, the product between the transposed matrix of the current normal vector and the reference vector is calculated as the first candidate vector, the product between the first candidate vector and the reference vector is calculated as the second candidate vector, and the current normal vector is subtracted from the second candidate vector to obtain the dissociated normal vector.

[0210] like Figure 10 As shown, the process of dissociating the normal vector is expressed as follows:

[0211]

[0212] Among them, n1 is the current normal vector, n2 is the reference vector, is the transposed matrix of the current normal vector, and n′1 is the normal vector after separation.

[0213] Step 806: For each facial attribute, adjust the latent vector along the normal vector to generate target facial data in the face reconstructor.

[0214] like Figure 9 As shown, after separating the normal vector that represents the facial attribute of wearing glasses from the normal vector that represents the facial attribute of age, the facial attribute that represents age of a certain original facial data (the facial data selected by the box) is edited. Going left means getting older, and going right means getting younger. In the target facial data after editing in the second row, the person will not only get older but also wear glasses.

[0215] Step 807: For each facial attribute, under the supervision of the paired original facial data and target facial data, adjust the face reconstructor to a face editor for editing facial attributes.

[0216] Example 3

[0217] Figure 11This is a flowchart of a face editing method provided in the first embodiment of the present invention. This embodiment is applicable to each situation. The method can be performed by a face editing device. The face editing device can be implemented by software and / or hardware and can be configured in a computer device, such as a server, workstation, personal computer, mobile terminal (such as a mobile phone, tablet computer, smart wearable device, etc.), etc., and specifically includes the following steps:

[0218] Step 1101: Receive original image data.

[0219] If this embodiment is applied to computer devices on user ends such as mobile terminals and wearable devices, applications that can perform image processing can be implemented in the computer devices, such as live broadcast applications, image editing applications, camera applications, instant messaging tools, gallery applications, etc.

[0220] Applications such as image editing applications, instant messaging tools, and gallery applications may have a UI (User Interface) that provides imported controls. Users can operate the imported controls through touch or mouse operations to select locally stored video data or image data (represented by thumbnails or paths), or select video data or image data stored on the network (represented by URLs (Uniform Resource Locators)).

[0221] Applications such as live streaming applications, image editing applications, camera applications, and instant messaging tools may have UIs that provide controls for taking photos and recording videos. Users can operate these controls through touch or mouse operations to notify the application to call the camera to collect video or image data.

[0222] If this embodiment is applied to non-user-end computer equipment such as servers and workstations, it can receive video data or image data sent by the user-end equipment.

[0223] For video data, one or more frames of image data may be extracted therefrom.

[0224] Face detection (Landmark) can be performed on these image data to determine whether there is a human figure and the area where the facial data is located. If there is a human figure, the area can be marked and cropped in the original image data, and the operation of editing facial attributes can continue. If there is no human figure, a prompt message can be generated to remind the user that there is no facial data and the operation of editing facial attributes can be stopped.

[0225] For the convenience of distinction, these image data can be recorded as original image data, that is, the original image data contains human images.

[0226] Among them, face detection, also known as facial key point detection, positioning or face alignment, refers to locating the key areas of the face, including eyebrows, eyes, nose, mouth, facial contours, etc., given facial data.

[0227] Face detection usually uses the following methods:

[0228] 1. Use artificially extracted features, such as Haar features, use the features to train a classifier, and use the classifier to perform face detection.

[0229] 2. Inherit face detection from general object detection algorithms, for example, using Faster R-CNN to detect faces.

[0230] 3. Use convolutional neural networks with cascade structures, for example, Cascade CNN (Cascade Convolutional Neural Networks) and MTCNN (Multi-task Cascaded Convolutional Networks).

[0231] In specific implementations, these methods for implementing face detection can be integrated into the application module, and the application can directly call the module to detect facial key points in the image data. These methods for implementing face detection can also be integrated into the SDK (Software Development Kit). The SDK serves as the assembly data of the application. The application can request the SDK to perform face detection on the image data. The SDK detects the facial key points in the image data and returns the facial key points to the application.

[0232] It should be noted that the number of facial key points can be set according to actual conditions. For static image processing, the real-time requirement is relatively low, and relatively dense facial key points, such as 1,000, can be detected. In addition to locating important facial feature points, the contours of the facial features can also be accurately described. For live broadcasts, etc., the real-time requirement is relatively high, and relatively sparse facial key points, such as 68, 81, and 106, can be detected to locate more obvious and important feature points on the face (such as eye key points, eyebrow key points, nose key points, mouth key points, contour key points, etc.) to reduce the processing volume and processing time, etc. The embodiments of the present invention are not limited to this.

[0233] It should be noted that for video data (such as live video data), there may be jitter between each frame of image data. During face detection, inter-frame smoothing can be performed to align the facial key points of the previous and next frames of image data. When cropping face data, inter-frame smoothing can be performed to align the facial key points of the previous and next image data frames.

[0234] Step 1102: Select a facial attribute for editing the character image from a plurality of facial attributes as a target attribute.

[0235] For multiple facial attributes, they can be encapsulated as interface elements in the form of filters, plug-ins, etc. The application in the user-side device loads these interface elements in the UI for the user to browse. The user can select one of the interface elements through touch operation or mouse operation. The facial attribute corresponding to the interface element is the facial attribute of the character image that the user currently wants to edit, and can be recorded as the target attribute.

[0236] Step 1103: Load a face editor for editing target attributes.

[0237] In this embodiment, multiple face editors can be pre-trained for multiple facial attributes, and each face editor can edit one facial attribute, that is, each facial attribute is configured with a face editor for editing the facial attribute.

[0238] It should be noted that the training method of the face editor can refer to any of the above-mentioned embodiments of the training method of the face editor, and this embodiment will not be described in detail here.

[0239] If the user selects a target attribute, the mapping relationship between the face attributes and the face editors recorded when the face editor is trained can be read, and the face editor mapped to the target attribute can be queried in the mapping relationship.

[0240] If this embodiment is applied to computer devices on the user side, such as mobile terminals and wearable devices, considering that there are a large number of facial attributes and a large number of face editors, in order to reduce the storage space occupied by the computer device, the server can push some face editors to the computer device in a certain way, or the server does not push the face editor to the computer device, but the user actively requests to download the face editor from the server.

[0241] For example, the server may sort the face editors according to their popularity and push the most popular face editors to the computer device.

[0242] For another example, the server may calculate a face editor preferred by the load user and push the face editor to the computer device.

[0243] The face editor is configured with identification information such as a number, and a query is made locally on the computer device to determine whether a face editor corresponding to the identification information is stored.

[0244] If the face editor corresponding to the identification information has been stored locally on the computer, the face editor and its parameters are loaded into the memory for execution.

[0245] If the face editor corresponding to the identification information is not stored locally on the computer, the face editor corresponding to the identification information can be requested from the server. The server sends the face editor to the computer device based on the request. The computer device records that the face editor corresponding to the identification information is stored locally, and loads the face editor and its parameters into the memory for execution.

[0246] Of course, the user can also instruct the computer device to delete the face editor stored locally. In this case, it is recorded that the face editor corresponding to the identification information is not stored locally.

[0247] If this embodiment is applied to non-user-end computer devices such as servers and workstations, the identification information such as the number configured by the face editor can be queried, the face editor corresponding to the identification information can be queried locally on the computer device, and the face editor and its parameters can be loaded into the memory for execution.

[0248] Step 1104: input the original image data into a face editor for editing to edit target attributes of the character image and obtain target image data.

[0249] The area containing the human image in the original image data is input into the face editor, and the face editor edits the target attributes of the human image. When the editing is completed, the target image data is output.

[0250] In a specific implementation, the face editor includes an encoder and a decoder. The area containing the human image in the original image data is input into the encoder, the original image data is encoded into a vector as a face vector, and the face vector is input into the decoder. The face vector is decoded into face data as the target image data.

[0251] Furthermore, the encoder and decoder are trained under the supervision of paired original face data and target face data on the basis of restoring the face data. In the space where the latent vector representing the facial attributes of the original face data is located, the latent vector is adjusted along the normal vector corresponding to the edited target attribute. The adjusted latent vector is used to decode into the target face data. Therefore, the original image data and the target image data satisfy the following relationship:

[0252] In the space where the latent vector representing the facial attributes of the original image data is located, the latent vector is adjusted along the normal vector corresponding to the edited target attribute, and the adjusted latent vector is used to decode into the target image data.

[0253] The target image data can be resized to obtain new target image data so that its size is consistent with the area of ​​the character shape, so that the target image data can be the background data of the original image data (the area except the character image).

[0254] In addition, if the original image data is image data in video data (such as live video data), face detection (Landmark) can be performed when adjusting the size, so as to perform inter-frame smoothing and align the facial key points of the previous and next image data.

[0255] Since the content of the reconstructed target image data may be different from the background data (area other than the character image) of the original image data, the character shape and the target image data can be fused to generate a new character shape so that the content of the target image data is the same or similar to that of the background data, and then the new character shape is pasted into the background data.

[0256] In this embodiment, original image data is received, which contains a human image. A facial attribute for editing the human image is selected from a plurality of facial attributes as a target attribute. Each facial attribute is configured with a face editor for editing the facial attribute. The face editor for editing the target attribute is loaded, and the original image data is input into the face editor for editing to edit the target attribute of the human image and obtain target image data. By independently setting a face editor for a single facial attribute, the face editor can maintain better performance while applying a simple structure, and the effect of editing the facial attribute is controllable. The structure of the face editor can be greatly simplified, and the number of samples used in training can be reduced. Moreover, the running of a face editor with a relatively simple structure consumes fewer resources and takes less time. It can be deployed on a higher-performance personal computer or cloud, or on a lower-performance mobile terminal. Editing facial attributes on the mobile terminal does not need to be uploaded to a personal computer or cloud for editing, which takes less time and has a faster response speed.

[0257] Example 4

[0258] Figure 12 This is a flowchart of a live broadcast method provided in the first embodiment of the present invention. This embodiment is applicable to each situation. The method can be performed by a live broadcast device. The live broadcast device can be implemented by software and / or hardware and can be configured in a computer device, such as a personal computer, a mobile terminal (such as a mobile phone, a tablet computer, a smart wearable device, etc.), etc., and specifically includes the following steps:

[0259] Step 1201: Collect video data.

[0260] Install applications with live broadcast and image processing functions on computer devices, such as independent live broadcast applications, instant messaging tools, short video applications, etc.

[0261] For these applications, the camera and microphone of the computer device can be called to collect video data and audio data, thereby synthesizing live video data.

[0262] There are multiple frames of image data in the video data, and some of the image data contain anchor users. Face detection can be performed on these image data to determine whether there is an area where the anchor user and his or her facial data are located. If there is a human image, the area can be marked and cropped in the image data, and the operation of editing facial attributes can continue.

[0263] For ease of distinction, these image data are recorded as original image data, that is, multiple frames of original image data in the video data contain the anchor user.

[0264] Step 1202: Select a facial attribute to be edited by the anchor user from a plurality of facial attributes as a target attribute.

[0265] For various facial attributes, they can be encapsulated as interface elements in the form of filters, plug-ins, etc., and these interface elements can be loaded in the application UI for users to browse. The user can select one of the interface elements through touch operation or mouse operation. The facial attribute corresponding to the interface element indicates that the host user currently wants to edit his facial attribute and can be recorded as the target attribute.

[0266] The target attributes can be pre-set based on factors such as the anchor user's business needs and the effects of the face editor. For example, the target attributes can include cartoonization (two-dimensional), female to male, aging, youth, European and American style, etc.

[0267] Step 1203: Load a face editor for editing target attributes.

[0268] In this embodiment, multiple face editors can be pre-trained for multiple facial attributes, and each face editor can edit one facial attribute, that is, each facial attribute is configured with a face editor for editing the facial attribute.

[0269] It should be noted that the training method of the face editor can refer to any of the above-mentioned embodiments of the training method of the face editor, and this embodiment will not be described in detail here.

[0270] If the user selects a target attribute, the mapping relationship between the face attributes and the face editors recorded when the face editor is trained can be read, and the face editor mapped to the target attribute can be queried in the mapping relationship.

[0271] Taking into account the situation that there are a large number of facial attributes and a large number of face editors, in order to reduce the storage space occupied by the computer device, the server can push some face editors to the computer device in a certain way, or the server does not push the face editor to the computer device, but the user actively requests to download the face editor from the server.

[0272] For example, the server may sort the face editors according to their popularity and push the most popular face editors to the computer device.

[0273] For another example, the server may calculate a face editor preferred by the load user and push the face editor to the computer device.

[0274] The face editor is configured with identification information such as a number, and a query is made locally on the computer device to determine whether a face editor corresponding to the identification information is stored.

[0275] If the face editor corresponding to the identification information has been stored locally on the computer, the face editor and its parameters are loaded into the memory for execution.

[0276] If the face editor corresponding to the identification information is not stored locally on the computer, the face editor corresponding to the identification information can be requested from the server. The server sends the face editor to the computer device based on the request. The computer device records that the face editor corresponding to the identification information is stored locally, and loads the face editor and its parameters into the memory for execution.

[0277] Of course, the user can also instruct the computer device to delete the face editor stored locally. In this case, it is recorded that the face editor corresponding to the identification information is not stored locally.

[0278] Step 1204: Input the original image data into the face editor for editing to edit the target attributes of the anchor user and obtain the target image data.

[0279] The area containing the anchor user in the original image data is input into the face editor, and the face editor edits the target attributes of the anchor user. When the editing is completed, the target image data is output.

[0280] In a specific implementation, the face editor includes an encoder and a decoder. The area containing the human image in the original image data is input into the encoder, the original image data is encoded into a vector as a face vector, and the face vector is input into the decoder. The face vector is decoded into face data as the target image data.

[0281] Furthermore, the encoder and decoder are trained under the supervision of paired original face data and target face data on the basis of restoring the face data. In the space where the latent vector representing the facial attributes of the original face data is located, the latent vector is adjusted along the normal vector corresponding to the edited target attribute. The adjusted latent vector is used to decode into the target face data. Therefore, the original image data and the target image data satisfy the following relationship:

[0282] In the space where the latent vector representing the facial attributes of the original image data is located, the latent vector is adjusted along the normal vector corresponding to the edited target attribute, and the adjusted latent vector is used to decode into the target image data.

[0283] The target image data can be resized to obtain new target image data so that its size is consistent with the area of ​​the anchor user, so that the target image data can be the background data of the original image data (the area except the character image).

[0284] In addition, if the original image data is image data in video data (such as live video data), face detection (Landmark) can be performed when adjusting the size, so as to perform inter-frame smoothing and align the facial key points of the previous and next image data.

[0285] Since the content of the reconstructed target image data may be different from the background data (areas other than the character image) of the original image data, the anchor user can be face-fused with the target image data to generate a new anchor user, so that the content of the target image data is the same or similar to that of the background data, and then the new anchor user is pasted into the background data.

[0286] Step 1205: Preview the video data containing the target image data, and stream the video data containing the target image data to the live broadcast room opened by the anchor user for publishing.

[0287] On the one hand, the application can be called to play video data containing target image data for the anchor user to browse and edit the effect of target attributes. Since the anchor user's target attributes in the video data are edited in the computer device, there is no need to transmit the video data. Therefore, the preview takes less time and the response speed is fast.

[0288] On the other hand, the video data containing the target image data can be uploaded to the live broadcast platform, and the live broadcast platform publishes the video data with the target image data in the live broadcast room opened by the anchor user. The client logged in by the audience user can enter the live broadcast room and play the video data.

[0289] In this embodiment, video data is collected, and multiple frames of raw image data in the video data contain a live broadcast user. A facial attribute to be edited for the live broadcast user is selected from multiple facial attributes as a target attribute. Each facial attribute is configured with a face editor for editing the facial attribute. The face editor for editing the target attribute is loaded, and the raw image data is input into the face editor for editing to edit the target attribute of the live broadcast user and obtain target image data. Video data containing the target image data is previewed, and the video data containing the target image data is streamed to a live broadcast room opened by the live broadcast user for publication. By independently setting a face editor for a single facial attribute, the face editor can maintain excellent performance while using a simple structure, and the effect of editing the facial attribute is controllable. This can greatly simplify the structure of the face editor and reduce the number of samples used in training. Furthermore, the face editor with a relatively simple running structure consumes fewer resources and takes less time. It can be deployed on mobile devices with lower performance. Facial attributes can be edited on the mobile device without uploading to a personal computer or the cloud for editing, resulting in less time and faster response.

[0290] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0291] Example 5

[0292] Figure 13 This is a structural block diagram of a face editor training device provided in Example 5 of the present invention, which may specifically include the following modules:

[0293] A face reconstruction training module 1301 is used to train a face reconstruction device for restoring a face using original face data as samples;

[0294] A latent vector training module 1302 is configured to train a latent vector for the original face data, wherein the latent vector represents a face attribute;

[0295] A normal vector search module 1303 is configured to search for a normal vector in the space where the latent vector is located, the normal vector indicating a direction for editing each of the facial attributes;

[0296] a target face data generating module 1304, configured to adjust the latent vector along the normal vector for each of the facial attributes to generate target face data in the face reconstructor;

[0297] The face reconstructor adjustment module 1305 is configured to adjust the face reconstructor into a face editor for editing the face attribute under the supervision of the paired original face data and the target face data for each face attribute.

[0298] In one embodiment of the present invention, the face reconstructor includes an encoder and a decoder; the face reconstructor training module 1301 is further configured to:

[0299] In the encoder, the original face data is encoded into a vector as a face vector;

[0300] In the decoder, the face vector is decoded into face data as reconstructed face data;

[0301] In the universal discriminator, a loss value is calculated for the original face data and the reconstructed face data as a reconstruction loss value;

[0302] Determining whether the reconstruction loss value converges;

[0303] If yes, it is determined that the face reconstructor training is completed;

[0304] If not, the decoder and the encoder are updated, and the original face data is input into the encoder for encoding to obtain a face vector.

[0305] In one example of the embodiments of the present invention, the encoder includes n convolutional layers and one fully connected layer, the dimension of the output vector of the convolutional layer ordered earlier is larger than the dimension of the output vector of the convolutional layer ordered later, the decoder is a comprehensive network in a style adversarial generative network, the comprehensive network includes n blocks, the dimension of the output vector of the block ordered earlier is smaller than the dimension of the output vector of the block ordered later, the universal discriminator includes n convolutional layers, and the n convolutional layers of the encoder have the same structure as the n convolutional layers of the discriminator;

[0306] The face reconstruction training module 1301 is further configured to:

[0307] In the encoder, calling n convolutional layers in sequence according to the order to perform convolution processing on the original face data to obtain a candidate vector;

[0308] Calling the fully connected layer to perform a fully connected operation on the candidate vector to obtain a 2n-layer face vector;

[0309] The face reconstruction training module 1301 is further configured to:

[0310] In the decoder, calling n blocks in sequence according to the order to process the 2n layers of face vectors to obtain reconstructed face data, wherein every two layers of face vectors are input to each block;

[0311] The face reconstruction training module 1301 is further configured to:

[0312] In the universal discriminator, a first universal facial feature is extracted from the original facial data, a second universal facial feature is extracted from the reconstructed facial data, and a loss value is calculated based on the first universal facial feature and the second universal facial feature as a reconstruction loss value.

[0313] In one embodiment of the present invention, the latent vector training module 1302 is further configured to:

[0314] Performing a loop operation, wherein the loop operation refers to an encoder in the face reconstructor encoding the original face data into a face vector, mapping the face vector into a latent vector according to a mapping parameter, and a decoder in the face reconstructor decoding the latent vector into new original face data;

[0315] Generate a loss value based on the loop operation as an attribute loss value;

[0316] Determining whether the attribute loss value converges;

[0317] If yes, it is determined that the latent vector training is completed;

[0318] If not, the mapping parameters are updated and the execution loop operation is returned.

[0319] In one example of an embodiment of the present invention, the latent vector training module 1302 is further configured to:

[0320] Calculating a first distance between the original face data and the new original face data;

[0321] Calculating a second distance representing a perceptual loss between the original face data and the new original face data;

[0322] Calculating a third distance between a latent vector corresponding to the original face data and a latent vector corresponding to the new original face data;

[0323] The first distance, the second distance, and the third distance are linearly fused into an attribute loss value.

[0324] In one embodiment of the present invention, the normal vector finding module 1303 is further configured to:

[0325] Randomly inputting the latent vector into the decoder of the face reconstructor for decoding to obtain multiple frames of sample face data;

[0326] For each facial attribute, performing binary classification on the facial data under the facial attribute to obtain the facial attribute and confidence level of the facial data;

[0327] Classifying part of the sample face data as positive samples and part of the sample face data as negative samples, wherein the confidence level of the positive samples is higher than the confidence level of the negative samples;

[0328] Determine a hyperplane in the space where the latent vector is located, where one side of the hyperplane includes the latent vector of the positive sample, and the other side of the hyperplane includes the latent vector of the negative sample;

[0329] The normal vector of the hyperplane is set to edit the normal vector of each of the facial attributes.

[0330] In one embodiment of the present invention, it further comprises:

[0331] A reference vector setting module, configured to set other normal vectors as reference vectors for the current normal vector, wherein the facial attributes edited by the current normal vector include the facial attributes edited by the reference vector;

[0332] The normal vector separation module is used to separate the reference vector from the current normal vector to obtain a separated normal vector.

[0333] In one embodiment of the present invention, the normal vector dissociation module is further configured to:

[0334] Calculate the transposed matrix of the current normal vector;

[0335] Calculate the product between the transposed matrix of the current normal vector and the reference vector as a first candidate vector;

[0336] Calculating the product between the first candidate vector and the reference vector as a second candidate vector;

[0337] The second candidate vector is subtracted from the current normal vector to obtain a normal vector after separation.

[0338] In one embodiment of the present invention, the target face data generation module 1304 is further configured to:

[0339] Determine editing intensity;

[0340] For each of the facial attributes, adjusting the normal vector corresponding to the facial attribute using the editing strength;

[0341] If the adjustment is completed, the normal vector is added to the latent vector to obtain the edited latent vector;

[0342] The adjusted latent vector is input into the decoder in the face reconstructor for decoding to obtain the target face data.

[0343] In one embodiment of the present invention, the face reconstructor adjustment module 1305 is further configured to:

[0344] For each of the facial attributes, inputting the original facial data into the face reconstructor to generate facial data as reference facial data;

[0345] Inputting the reference face data of multiple resolutions and the target face data of multiple resolutions into multiple personality discriminators respectively to extract a first personality facial feature of the reference face data, extract a second personality facial feature of the target face data, and output a true or false discrimination result;

[0346] Calculating a loss value of editing the facial attributes by the face reconstructor based on the discrimination result as an adversarial loss value;

[0347] Calculating a feature matching loss value based on the first individual facial feature and the second individual facial feature;

[0348] Calculating a perception loss value based on the target face data and the reference face data;

[0349] Linearly fusing the adversarial loss value, the feature matching loss value, and the perceptual loss value into a total loss value;

[0350] Under the constraint of maximizing the adversarial loss value, determining whether the total loss value converges;

[0351] If so, determining that the face reconstructor has been adjusted to a face editor for editing the face attributes;

[0352] If not, the decoder and the encoder are updated, and the process returns to executing the process for each facial attribute, inputting the original facial data into the face reconstructor to generate facial data as reference facial data.

[0353] The training device for the face editor provided in the embodiment of the present invention can execute the training method for the face editor provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0354] Example 6

[0355] Figure 14 This is a structural block diagram of a face editing device provided in Example 6 of the present invention, which may specifically include the following modules:

[0356] The original image data receiving module 1401 is used to receive original image data, wherein the original image data contains a human image;

[0357] A target attribute selection module 1402 is configured to select a facial attribute to be edited for the character image from a plurality of facial attributes as a target attribute, wherein each facial attribute is configured with a face editor for editing the facial attribute;

[0358] A face editor loading module 1403 is used to load a face editor for editing the target attributes;

[0359] The face attribute editing module 1404 is used to input the original image data into the face editor for editing, so as to edit the target attributes of the character image and obtain target image data.

[0360] In one embodiment of the present invention, the face editor loading module 1403 includes:

[0361] A face reconstruction module training module is used to train a face reconstruction module for restoring faces using original face data as samples;

[0362] A latent vector training module, configured to train a latent vector for representing facial attributes on the original facial data;

[0363] A normal vector search module, configured to search for a normal vector for editing each of the facial attributes in the space where the latent vector is located;

[0364] a target face data generating module, configured to adjust the latent vector along the normal vector for each of the facial attributes to generate target face data in the face reconstructor;

[0365] A face reconstructor adjustment module is used to adjust the face reconstructor into a face editor for editing the face attribute under the supervision of the paired original face data and the target face data for each face attribute.

[0366] In one embodiment of the present invention, the face editor loading module 1403 further includes:

[0367] A reference vector setting module, configured to set other normal vectors as reference vectors for the current normal vector, wherein the facial attributes edited by the current normal vector include the facial attributes edited by the reference vector;

[0368] The normal vector separation module is used to separate the reference vector from the current normal vector to obtain a separated normal vector.

[0369] The face editing device provided by the embodiment of the present invention can execute the face editing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0370] Example 7

[0371] Figure 15 This is a structural block diagram of a live broadcast device provided in Embodiment 7 of the present invention, which may specifically include the following modules:

[0372] The video data acquisition module 1501 is used to acquire video data, wherein multiple frames of original image data in the video data contain the anchor user;

[0373] A target attribute selection module 1502 is configured to select a facial attribute to be edited by the anchor user from a plurality of facial attributes as a target attribute, wherein each facial attribute is configured with a face editor for editing the facial attribute;

[0374] A face editor loading module 1503 is used to load a face editor for editing the target attributes;

[0375] The target face data generating module 1504 is configured to input the original image data into the face editor for editing, so as to edit the target attributes of the anchor user and obtain target image data;

[0376] The live broadcast processing module 1505 is used to preview the video data containing the target image data, and to push the video data containing the target image data to the live broadcast room opened by the anchor user for publishing.

[0377] In one embodiment of the present invention, the face editor loading module 1503 includes:

[0378] A face reconstruction module training module is used to train a face reconstruction module for restoring faces using original face data as samples;

[0379] A latent vector training module, configured to train a latent vector for representing facial attributes on the original facial data;

[0380] A normal vector search module, configured to search for a normal vector for editing each of the facial attributes in the space where the latent vector is located;

[0381] a target face data generating module, configured to adjust the latent vector along the normal vector for each of the facial attributes, so as to generate target face data in the face reconstructor;

[0382] A face reconstructor adjustment module is used to adjust the face reconstructor into a face editor for editing the face attribute under the supervision of the paired original face data and the target face data for each face attribute.

[0383] In one embodiment of the present invention, the face editor loading module 1503 further includes:

[0384] A reference vector setting module, configured to set other normal vectors as reference vectors for the current normal vector, wherein the facial attributes edited by the current normal vector include the facial attributes edited by the reference vector;

[0385] The normal vector separation module is used to separate the reference vector from the current normal vector to obtain a separated normal vector.

[0386] The live broadcast device provided in the embodiment of the present invention can execute the live broadcast method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0387] Example 8

[0388] Figure 16 A schematic diagram of the structure of a computer device provided in Example 8 of the present invention. Figure 16 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 16 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0389] like Figure 16 As shown, computer device 12 is implemented as a general-purpose computing device. Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0390] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0391] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0392] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 16 Not shown, often called a "hard drive"). Although Figure 16 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0393] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally implement the functions and / or methods of the embodiments described herein.

[0394] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0395] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the face editor training method, face editing method or live broadcast method provided in the embodiment of the present invention.

[0396] Example 9

[0397] Embodiment 9 of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the various processes of the above-mentioned face editor training method, face editing method, or live broadcast method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0398] Among them, computer-readable storage media can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0399] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A training method for a face editor, characterized in that: include: Using original face data as samples, a face reconstruction tool is trained to restore faces. Training the original face data to generate a latent vector for characterizing face attributes, including: encoding the original face data into a face vector by an encoder in the face reconstructor, mapping the face vector into a latent vector according to a mapping parameter, and decoding the latent vector into new original face data by a decoder in the face reconstructor; calculating a first distance between the original face data and the new original face data; calculating a second distance between the original face data and the new original face data that characterizes a perceptual loss; calculating a third distance between the latent vector corresponding to the original face data and the latent vector corresponding to the new original face data; and linearly fusing the first distance, the second distance, and the third distance into an attribute loss value; Determining whether the attribute loss value converges; if so, determining that the latent vector training is complete; if not, updating the mapping parameters and returning to the step of training the latent vector for representing facial attributes on the original face data; Finding a normal vector in the space where the latent vector is located, where the normal vector represents a direction for editing each of the facial attributes; For each of the facial attributes, adjusting the latent vector along the normal vector to generate target facial data in the face reconstructor; For each of the facial attributes, under the supervision of the paired original facial data and the target facial data, the face reconstructor is adjusted to a face editor for editing the facial attribute.

2. The method according to claim 1, characterized in that The face reconstructor includes an encoder and a decoder; The method of training a face reconstruction device for restoring a face using the original face data as a sample includes: In the encoder, the original face data is encoded into a vector as a face vector; In the decoder, the face vector is decoded into face data as reconstructed face data; In the universal discriminator, a loss value is calculated for the original face data and the reconstructed face data as a reconstruction loss value; Determining whether the reconstruction loss value converges; If yes, it is determined that the face reconstructor training is completed; If not, the decoder and the encoder are updated, and the original face data is input into the encoder for encoding to obtain a face vector.

3. The method according to claim 2, characterized in that The encoder includes n convolutional layers and one fully connected layer, the dimension of the output vector of the convolutional layer that is sorted first is larger than the dimension of the output vector of the convolutional layer that is sorted later, the decoder is a comprehensive network in a style adversarial generative network, the comprehensive network includes n blocks, the dimension of the output vector of the block that is sorted first is smaller than the dimension of the output vector of the block that is sorted later, and the universal discriminator includes n convolutional layers, and the n convolutional layers of the encoder have the same structure as the n convolutional layers of the discriminator; In the encoder, encoding the original face data into a vector as a face vector includes: In the encoder, calling n convolutional layers in sequence according to the order to perform convolution processing on the original face data to obtain a candidate vector; Calling the fully connected layer to perform a fully connected operation on the candidate vector to obtain a 2n-layer face vector; The decoding of the face vector into face data as reconstructed face data in the decoder includes: In the decoder, calling n blocks in sequence according to the order to process the 2n layers of face vectors to obtain reconstructed face data, wherein every two layers of face vectors are input to each block; In the universal discriminator, calculating a loss value for the original face data and the reconstructed face data as a reconstruction loss value includes: In the universal discriminator, a first universal facial feature is extracted from the original facial data, a second universal facial feature is extracted from the reconstructed facial data, and a loss value is calculated based on the first universal facial feature and the second universal facial feature as a reconstruction loss value.

4. The method according to claim 1, wherein The step of searching for a normal vector in the space where the latent vector is located includes: Randomly inputting the latent vector into the decoder of the face reconstructor for decoding to obtain multiple frames of sample face data; For each facial attribute, performing binary classification on the facial data under the facial attribute to obtain the facial attribute and confidence level of the facial data; Classifying part of the sample face data as positive samples and part of the sample face data as negative samples, wherein the confidence level of the positive samples is higher than the confidence level of the negative samples; Determine a hyperplane in the space where the latent vector is located, where one side of the hyperplane includes the latent vector of the positive sample, and the other side of the hyperplane includes the latent vector of the negative sample; The normal vector of the hyperplane is set to edit the normal vector of each of the facial attributes.

5. The method according to claim 1, wherein After finding the normal vector in the space where the latent vector is located, the following steps are further included: For the current normal vector, setting the other normal vectors as reference vectors, wherein the facial attributes edited by the current normal vector include the facial attributes edited by the reference vector; The reference vector is separated from the current normal vector to obtain a separated normal vector.

6. The method according to claim 5, characterized in that The separating the reference vector from the current normal vector to obtain a separated normal vector includes: Calculate the transposed matrix of the current normal vector; Calculate the product between the transposed matrix of the current normal vector and the reference vector as a first candidate vector; Calculating the product between the first candidate vector and the reference vector as a second candidate vector; The second candidate vector is subtracted from the current normal vector to obtain a normal vector after separation.

7. The method according to any one of claims 1 to 6, characterized in that The step of adjusting the latent vector along the normal vector for each facial attribute to generate target facial data in the face reconstructor includes: Determine editing intensity; For each of the facial attributes, adjusting the normal vector corresponding to the facial attribute using the editing strength; If the adjustment is completed, the normal vector is added to the latent vector to obtain the edited latent vector; The adjusted latent vector is input into the decoder in the face reconstructor for decoding to obtain the target face data.

8. The method according to any one of claims 1 to 6, characterized in that For each of the facial attributes, under the supervision of the paired original facial data and the target facial data, adjusting the face reconstructor to a face editor for editing the facial attribute comprises: For each of the facial attributes, inputting the original facial data into the face reconstructor to generate facial data as reference facial data; Inputting the reference face data of multiple resolutions and the target face data of multiple resolutions into multiple personality discriminators respectively to extract a first personality facial feature of the reference face data, extract a second personality facial feature of the target face data, and output a true or false discrimination result; Calculating a loss value of editing the facial attributes by the face reconstructor based on the discrimination result as an adversarial loss value; Calculating a feature matching loss value based on the first individual facial feature and the second individual facial feature; Calculating a perception loss value based on the target face data and the reference face data; Linearly fusing the adversarial loss value, the feature matching loss value, and the perceptual loss value into a total loss value; Under the constraint of maximizing the adversarial loss value, determining whether the total loss value converges; If so, determining that the face reconstructor has been adjusted to a face editor for editing the face attributes; If not, the decoder and the encoder are updated, and the process returns to executing the process for each facial attribute, inputting the original facial data into the face reconstructor to generate facial data as reference facial data.

9. A face editing method, characterized in that: include: receiving original image data, wherein the original image data includes a human image; Selecting a facial attribute for editing the character image from a plurality of facial attributes as a target attribute, each of the facial attributes being configured with a face editor for editing the facial attribute; loading a face editor for editing the target attributes; Inputting the original image data into the face editor for editing, so as to edit the target attributes of the character image and obtain target image data; The face editor is obtained according to the training method of the face editor according to any one of claims 1-8.

10. A live broadcast method, characterized in that: include: Collecting video data, wherein multiple frames of original image data in the video data contain a host user; Selecting a facial attribute to be edited by the anchor user from a plurality of facial attributes as a target attribute, each of the facial attributes being configured with a face editor for editing the facial attribute; loading a face editor for editing the target attributes; Inputting the original image data into the face editor for editing to edit the target attributes of the anchor user and obtain target image data; Previewing the video data containing the target image data, and streaming the video data containing the target image data to a live broadcast room opened by the anchor user for publishing; The face editor is obtained according to the training method of the face editor according to any one of claims 1-8.

11. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the face editor as described in any one of claims 1-8, or the face editing method as described in claim 9, or the live broadcast method as described in claim 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the training method of the face editor as described in any one of claims 1 to 8, or the face editing method as described in claim 9, or the live broadcast method as described in claim 10.

Images