Training and generation method of avatar generation model in American comic style and electronic device

Abstract

This invention relates to the field of image processing technology, specifically to a training method and electronic device for a comic book-style avatar generation model. The training method includes acquiring sample image pairs, which include real avatars and their corresponding cartoon avatars; inputting the real avatars into the feature encoding module of the avatar generation model to determine feature encoding; inputting the feature encodings into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar, the avatar style generation module being pre-trained; detecting facial key points between the cartoon avatar and the predicted cartoon avatar, and determining the facial key point loss; determining the image transformation loss based on the difference between the cartoon avatar and the preset cartoon avatar; and updating the parameters of the feature encoding module based on the fusion result of the facial key point loss and the image transformation loss to determine the target avatar generation model. This method adds a dimension for examining target consistency, resulting in faster convergence to the target effect during image transformation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to the training, generation method, and electronic device for generating comic book-style avatars. Background Technology

[0002] Existing avatar generation models can randomly generate images, such as faces and anime characters. Artificial neural networks can generate a generator that transforms a discrete latent space into a face image. The latent space refers to the space containing the compressed or encoded data (i.e., feature vectors) of the original data. However, editing real face images requires first projecting the image into the latent space to generate hidden codes, and then reconstructing the face based on these codes. This reconstruction process is time-consuming and yields unsatisfactory results. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a training and generation method and electronic device for an American comic-style avatar generation model, in order to solve the problems of low efficiency and unsatisfactory results in avatar generation.

[0004] According to a first aspect, embodiments of the present invention provide a training method for a comic book-style avatar generation model, comprising:

[0005] Obtain sample image pairs, which include real headshots and corresponding cartoon headshots;

[0006] The real headshot is input into the feature encoding module of the headshot generation model to determine the feature encoding;

[0007] The feature encoding is input into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar. The avatar style generation module is pre-trained.

[0008] Detect the facial landmarks of the cartoon avatar and the predicted cartoon avatar, and determine the facial landmark loss;

[0009] Based on the difference between the cartoon avatar and the predicted cartoon avatar, the image conversion loss is determined;

[0010] Based on the fusion result of the facial key point loss and the image transformation loss, the parameters of the feature encoding module are updated to determine the target avatar generation model.

[0011] The training method for the American comic-style avatar generation model provided in this embodiment of the invention adds facial key point loss to the loss function design, and correspondingly increases the dimension of target consistency examination. In image conversion, it converges to the target effect faster, improving training efficiency and the accuracy of the obtained target avatar generation model. At the same time, since the avatar style generation module obtained by pre-training is used in the training process, the training time of the model is reduced.

[0012] In some implementations, updating the parameters of the feature encoding module based on the fusion result of the facial landmark loss and the image transformation loss to determine the target avatar generation model includes:

[0013] Obtain multiple sets of weight combinations, wherein the weight combinations include weights that correspond one-to-one with the facial key point loss and the image transformation loss;

[0014] For each of the weight combinations, the facial key point loss and the image transformation loss are fused to obtain the corresponding comprehensive loss;

[0015] The parameters of the feature encoding module are updated using the comprehensive loss to obtain an optional avatar generation model;

[0016] Based on the training performance of the optional avatar generation models, the target avatar generation model is determined from the optional avatar generation models, wherein the training performance includes at least one of training time and accuracy of the optional avatar generation models.

[0017] The training method for the American comic-style avatar generation model provided in this invention uses a multi-weight combination approach when fusing facial key point loss and image transformation loss during the training process. That is, multiple different weights are used for fusion, and the target avatar generation model is determined by using the training effect of the optional avatar generation model obtained by each weight combination, thereby improving the accuracy of the target avatar generation model.

[0018] In some implementations, obtaining multiple sets of weight combinations includes:

[0019] Obtain the value range of each of the weights;

[0020] The values ​​of the weights are generated based on the range of values ​​to obtain the multiple sets of weight combinations.

[0021] The training method for the comic book-style avatar generation model provided in this embodiment of the invention determines the value of each weight by utilizing the value range of each weight, ensuring that the obtained weight combinations can meet the needs of practical applications.

[0022] In some implementations, determining the image transformation loss based on the difference between the cartoon avatar and the predicted cartoon avatar includes:

[0023] Facial features are extracted from the cartoon avatar and the predicted cartoon avatar respectively to obtain the first facial feature and the second facial feature;

[0024] Loss is calculated based on the first facial feature and the second facial feature to determine the feature loss;

[0025] Based on the cartoon avatar and the predicted cartoon avatar, pixel loss and perceptual loss are calculated respectively. The image transformation loss includes the feature loss, the pixel loss and the perceptual loss.

[0026] The training method for the comic-style avatar generation model provided in this invention includes feature loss, pixel loss, and perceptual loss in the image transformation loss. That is, multiple losses are combined during the training process to further ensure the accuracy of the target avatar generation model obtained from the training.

[0027] In some implementations, the feature encoding module is built based on a feature pyramid network, and the step of inputting the real avatar into the feature encoding module of the avatar generation model to determine the feature encoding includes:

[0028] The real headshot is input into the feature encoding module, and multiple pyramid-scale feature vectors are extracted to determine the feature encoding.

[0029] In some implementations, inputting the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar includes:

[0030] Based on the pyramid scale of each feature vector in the feature encoding, the feature vector is input into the corresponding network layer of the avatar style generation module for image reconstruction to determine the predicted cartoon avatar.

[0031] The training method for the comic book-style avatar generation model provided in this invention is based on a feature pyramid network structure. It extracts style vectors from different pyramid scales and directly inserts them into a fixed pre-trained avatar style generation module according to their spatial scale. This can directly complete image reconstruction and generate the transformed image, avoiding the time-consuming optimization process.

[0032] According to a second aspect, embodiments of the present invention provide a method for generating comic book-style avatars, including:

[0033] Obtain the face image to be processed;

[0034] The face image to be processed is input into the target avatar generation model to obtain the target cartoon avatar. The target avatar generation model is trained by the training method of the American comic style avatar generation model according to the first aspect or any embodiment of the first aspect of the present invention.

[0035] The American comic-style avatar generation method provided in this embodiment of the invention, due to the accuracy of the trained target avatar generation model, produces a more closely matching target cartoon avatar after processing the face image to be processed, and the entire processing has a shorter delay, enabling the rapid generation of the target cartoon avatar corresponding to the face image to be processed.

[0036] In some implementations, acquiring the face image to be processed includes:

[0037] Obtain the image to be processed, which contains human faces;

[0038] Face detection is performed on the image to be processed to obtain a face image;

[0039] Based on the face image, face alignment and normalization processing are performed to determine the face image to be processed.

[0040] The method for generating comic-style avatars provided in this invention performs face alignment and normalization processing after detecting the face image in the image to be processed. This ensures that the image to be processed input to the target avatar generation model meets the size requirements of the model, and that face alignment enables more accurate recognition of facial features, thereby further improving the efficiency and reliability of the generated target cartoon avatars.

[0041] According to a third aspect, embodiments of the present invention provide a training apparatus for a comic book-style avatar generation model, comprising:

[0042] The first acquisition module is used to acquire sample image pairs, the sample image pairs including real headshots and corresponding cartoon headshots;

[0043] The first input module is used to input the real avatar into the feature encoding module of the avatar generation model to determine the feature encoding;

[0044] The second input module is used to input the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar. The avatar style generation module is pre-trained.

[0045] The detection module is used to detect the facial key points of the cartoon avatar and the predicted cartoon avatar, and to determine the facial key point loss;

[0046] The determination module is used to determine the image conversion loss based on the difference between the cartoon avatar and the predicted cartoon avatar;

[0047] An update module is used to update the parameters of the feature encoding module based on the fusion result of the facial key point loss and the image transformation loss, so as to determine the target avatar generation model.

[0048] According to a fourth aspect, embodiments of the present invention provide an apparatus for generating comic book-style avatars, comprising:

[0049] The second acquisition module is used to acquire the face image to be processed;

[0050] The third input module is used to input the face image to be processed into the target avatar generation model to obtain the target cartoon avatar. The target avatar generation model is trained by the training method of the American comic style avatar generation model according to the first aspect of the present invention or any embodiment of the first aspect.

[0051] According to a fifth aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform a training method for a comic book-style avatar generation model as described in the first aspect or any embodiment of the first aspect, or to perform a comic book-style avatar generation method as described in the second aspect or any embodiment of the second aspect.

[0052] According to a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing the computer to execute the training method for the comic book-style avatar generation model described in the first aspect or any embodiment of the first aspect, or to execute the comic book-style avatar generation method described in the second aspect or any embodiment of the second aspect.

[0053] For the beneficial effects of the training device, device, electronic device and computer-readable storage medium for the comic book-style avatar generation model provided in this embodiment of the invention, please refer to the description of the training method and the corresponding beneficial effects of the comic book-style avatar generation method above, which will not be repeated here. Attached Figure Description

[0054] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0055] Figure 1 This is a flowchart of a training method for a comic book-style avatar generation model according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart of a training method for a comic book-style avatar generation model according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of the target avatar generation model according to an embodiment of the present invention;

[0058] Figure 4 This is a flowchart of a method for generating comic book-style avatars according to an embodiment of the present invention;

[0059] Figure 5 This is a structural block diagram of a training device for generating an American comic-style avatar according to an embodiment of the present invention;

[0060] Figure 6 This is a structural block diagram of a comic book-style avatar generation device according to an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] According to embodiments of the present invention, a training method for a comic book-style avatar generation model is provided, as well as an embodiment of the comic book-style avatar generation method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0064] This embodiment provides a training method for a comic book-style avatar generation model, which can be used in electronic devices such as computers, servers, and mobile terminals. Figure 1 This is a flowchart of a training method for a comic book-style avatar generation model according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0065] S11, Obtain sample image pairs.

[0066] The sample image pairs include real headshots and corresponding cartoon headshots.

[0067] Real-life portraits and their corresponding cartoon avatars can be of any age, gender, and skin color. To ensure that the target avatar generation model trained subsequently can be applied to different application scenarios, the age, gender, skin color, etc., of the faces can be unrestricted when selecting sample image pairs. Sample image pairs can be stored on electronic devices or in cloud storage. The electronic device can retrieve sample image pairs from the cloud storage when training the target avatar generation model, and so on. The specific settings depend on actual needs; there are no restrictions on the source of the sample image pairs.

[0068] S12, Input the real headshot into the feature encoding module of the headshot generation model to determine the feature encoding.

[0069] The avatar generation model includes a feature encoding module and an avatar style generation module. The feature encoding module encodes the features of the input real avatar to generate an extended latent space code. Here, the latent space refers to the space where the original data is compressed (encoded) (i.e., feature vectors), and the latent code is the feature vector of the latent space.

[0070] S13, input the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar.

[0071] The avatar style generation module is pre-trained.

[0072] The avatar style generation model takes feature encoding as input and outputs predicted cartoon avatars. The model reconstructs the input feature encoding; its network structure can be based on the StyleGAN generator or other network models, without any restrictions here.

[0073] S14, detect and predict the facial landmarks of the cartoon avatar and determine the facial landmark loss.

[0074] For detecting facial landmarks in cartoon avatars, the facial landmarks of the cartoon avatars can be determined and stored together with the cartoon avatars when constructing sample image pairs. The detection of facial landmarks can be achieved using a facial landmark detection model, such as key point extraction based on a lightweight PFLD network.

[0075] After obtaining the cartoon avatar and predicting its facial landmarks, the loss function for the facial landmarks is calculated using an appropriate loss function. For example, Wing loss can be used to calculate the loss function.

[0076] S15, based on the difference between the cartoon avatar and the predicted cartoon avatar, determine the image transformation loss.

[0077] The difference between the cartoon avatar and the predicted cartoon avatar can be determined by the difference in pixels at the same location, or by the facial features of both avatars, with feature loss calculated based on facial features; or by fusing multiple losses to obtain the image transformation loss.

[0078] In some implementations, the image conversion loss may also include face shape and expression loss. For example, the avatar generation model in this embodiment is based on a 3D face deformation statistical model, which reconstructs a face from a 2D image into a 3D image. By solving the 3D face reconstruction process, the face shape, texture, and expression parameters can be accurately quantified to obtain a parameter sequence of relevant face attributes. In the generation process of the cartoonized face image in this embodiment, the 3D face attribute parameters of both the predicted cartoon face and the target cartoon face can be calculated simultaneously. Furthermore, based on the mean squared error criterion, the face shape and expression loss can be calculated, effectively making the cartoon face generated by the avatar generation model in this embodiment approximate the real target in terms of facial expression and other attributes.

[0079] The specifics of this step will be described in detail below.

[0080] S16. Based on the fusion result of facial landmark loss and image transformation loss, the parameters of the feature encoding module are updated to determine the target avatar generation model.

[0081] The electronic device fuses the facial landmark loss and image transformation loss to obtain a comprehensive loss; then, based on this comprehensive loss, the parameters of the feature encoding module are updated, and finally the target avatar generation model is determined.

[0082] The specifics of this step will be described in detail below.

[0083] The training method for the comic book-style avatar generation model provided in this embodiment adds facial key point loss to the loss function design, and correspondingly increases the dimension of target consistency assessment. In image conversion, it converges to the target effect more quickly, improving training efficiency and the accuracy of the obtained target avatar generation model. At the same time, since the avatar style generation module obtained through pre-training is used in the training process, the training time of the model is reduced.

[0084] This embodiment provides a training method for a comic book-style avatar generation model, which can be used in electronic devices such as computers, servers, and mobile terminals. Figure 2 This is a flowchart of a training method for a comic book-style avatar generation model according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0085] S21, Obtain sample image pairs.

[0086] The sample image pairs include real headshots and corresponding cartoon headshots.

[0087] Please see details Figure 1 S11 of the illustrated embodiment will not be described again here.

[0088] S22, Input the real headshot into the feature encoding module of the headshot generation model to determine the feature encoding.

[0089] In some implementations, such as Figure 3 As shown, the feature encoding module is constructed based on a feature pyramid network. Therefore, S22 above includes: inputting the real headshot into the feature encoding module, extracting feature vectors at multiple pyramid scales to determine the feature encoding.

[0090] S23, input the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar.

[0091] The avatar style generation module is pre-trained.

[0092] In some implementations, S23 includes: inputting the feature vectors into the corresponding network layer of the avatar style generation module for image reconstruction based on the pyramid scale of each feature vector in the feature encoding, thereby determining the predicted cartoon avatar. For example, the feature encoding module maps the real image to an extended latent space w+ (18*512*1, 18 different 512-dimensional w vectors). Based on the feature pyramid network structure, style vectors are extracted from different pyramid scales and directly inserted into a fixed pre-trained avatar style generation module according to their spatial scale, which can directly complete image reconstruction and generate the transformed image.

[0093] Based on the feature pyramid network structure, style vectors are extracted from different pyramid scales and directly inserted into a fixed pre-trained avatar style generation module according to their spatial scale. This can directly complete image reconstruction and generate the transformed image, avoiding the time-consuming optimization process.

[0094] S24, detect and predict the facial landmarks of the cartoon avatar and determine the facial landmark loss.

[0095] Please see details Figure 1 S14 of the illustrated embodiment will not be described again here.

[0096] S25. Based on the difference between the cartoon avatar and the predicted cartoon avatar, determine the image transformation loss.

[0097] Specifically, S25 includes:

[0098] S251, extract facial features from the cartoon avatar and the predicted cartoon avatar respectively to obtain the first facial feature and the second facial feature.

[0099] When electronic devices extract facial features, they use facial feature extraction models or perform image feature analysis on the corresponding images to obtain first facial features and second facial features. The first facial feature corresponds to a cartoon avatar, and the second facial feature corresponds to a predicted cartoon avatar.

[0100] S252, calculate the loss based on the first face feature and the second face feature to determine the feature loss.

[0101] Electronic devices can use corresponding loss functions to calculate feature loss, which include, but are not limited to, L1 loss, L2 loss, etc.

[0102] S253 calculates pixel loss and perceptual loss based on the cartoon avatar and the predicted cartoon avatar, respectively.

[0103] The image transformation loss includes feature loss, pixel loss, and perceptual loss.

[0104] Pixel loss can be achieved using the least mean square loss function, which calculates the difference between the pixels at the corresponding positions of the cartoon head and the predicted cartoon head to obtain the pixel loss; perceptual loss is used to represent the loss due to the conversion of one image into another.

[0105] S26. Based on the fusion result of facial landmark loss and image transformation loss, the parameters of the feature encoding module are updated to determine the target avatar generation model.

[0106] Specifically, S26 above includes:

[0107] S261, obtain multiple weight combinations.

[0108] The weight combination includes weights that correspond one-to-one with the facial landmark loss and the image transformation loss.

[0109] Each loss corresponds to a weight. For example, if the loss consists of 3 losses, then each weight combination includes 3 weights.

[0110] In some embodiments, S261 includes:

[0111] (1) Obtain the value range of each weight.

[0112] (2) Generate weight values ​​based on the value range to obtain multiple weight combinations.

[0113] The values ​​of each weight can be a continuous range of variation, a discrete range of variation, and so on. When determining the values ​​of each weight, values ​​are selected from either continuous or discrete ranges of variation, thus obtaining multiple sets of weight combinations.

[0114] The values ​​of each weight are determined by using the range of values ​​for each weight, ensuring that the resulting weight combinations can meet the needs of practical applications.

[0115] S262, for each weight combination, the facial landmark loss and image transformation loss are fused to obtain the corresponding comprehensive loss.

[0116] When calculating the comprehensive loss, the weights of each weight combination and the corresponding loss are weighted and summed to achieve the fusion of facial landmark loss and image transformation loss, thus obtaining the comprehensive loss.

[0117] S263 uses the comprehensive loss to update the parameters of the feature encoding module, resulting in an optional avatar generation model.

[0118] When training the avatar generation model, the comprehensive weights are calculated using each set of weight combinations. After training, optional avatar generation models corresponding one-to-one with each set of weight combinations are obtained. Furthermore, when determining the optional avatar generation models, the training time of each model is recorded. Alternatively, the accuracy of the optional avatar generation models is tested using a test dataset to obtain the accuracy of each model.

[0119] S264, based on the training effect of the optional avatar generation models, determines the target avatar generation model from the optional avatar generation models.

[0120] The training effect includes at least one of the training duration and the accuracy of the optional avatar generation model.

[0121] When determining the target avatar generation model, the various values ​​of the training effect can be combined to obtain each optional avatar generation model. For example, the training time and accuracy can be normalized to obtain the training effect value of each optional avatar generation model, and the optional avatar generation model with the highest training effect can be determined as the target avatar generation model.

[0122] Alternatively, other methods can be used to determine the target avatar generation model, and no restrictions are imposed on them here.

[0123] The training method for the comic-style avatar generation model provided in this embodiment includes feature loss, pixel loss, and perceptual loss in the image transformation loss. That is, multiple losses are combined during the training process to further ensure the accuracy of the target avatar generation model. When fusing the facial keypoint loss and image transformation loss during training, a multi-set weight combination method is used. That is, multiple different sets of weights are used for fusion. Finally, the training effect of the optional avatar generation models obtained by each set of weight combinations is used to determine the target avatar generation model, thereby improving the accuracy of the target avatar generation model.

[0124] This embodiment provides a method for generating comic book-style avatars, which can be used on electronic devices, such as mobile terminals. Figure 4 This is a flowchart of a training method for a comic book-style avatar generation model according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0125] S31, Obtain the face image to be processed.

[0126] The facial images to be processed can be uploaded by the user through interaction with the electronic device, stored in the electronic device, or captured in real time using the image acquisition device of the electronic device, and so on.

[0127] In some embodiments, S31 includes:

[0128] (1) Obtain the image to be processed containing human faces.

[0129] (2) Perform face detection on the image to be processed to obtain a face image.

[0130] (3) Perform face alignment and normalization processing based on the face image to determine the face image to be processed.

[0131] The electronic device takes any image containing a face, performs face detection, face alignment, and normalization to a preset size to determine the face image to be processed. After detecting the face image in the image to be processed, face alignment and normalization processing ensure that the image to be processed input to the target avatar generation model meets the model's size requirements, and that face alignment enables more accurate recognition of facial features, further improving the efficiency and reliability of the generated target cartoon avatar.

[0132] S32, input the face image to be processed into the target avatar generation model to obtain the target cartoon avatar.

[0133] The target avatar generation model is trained according to the training method of the comic book style avatar generation model described in any of the above embodiments. For the specific structural details of the target avatar generation model, please refer to the above description, which will not be repeated here.

[0134] The electronic device inputs the face image to be processed into the target avatar generation model, and uses the target avatar generation model to process the face image to obtain the target cartoon avatar, that is, to generate a cartoon-style image of the corresponding face.

[0135] The comic book-style avatar generation method provided in this embodiment, due to the accuracy of the trained target avatar generation model, produces a more closely matching target cartoon avatar after processing the face image to be processed, and the entire processing has a shorter delay, enabling the rapid generation of the target cartoon avatar corresponding to the face image to be processed.

[0136] This embodiment also provides a training device for a comic book-style avatar generation model and a comic book-style avatar generation device. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0137] This embodiment provides a training device for generating a comic book-style avatar model, such as... Figure 5 As shown, it includes:

[0138] The first acquisition module 41 is used to acquire sample image pairs, the sample image pairs including real headshots and corresponding cartoon headshots;

[0139] The first input module 42 is used to input the real avatar into the feature encoding module of the avatar generation model to determine the feature encoding;

[0140] The second input module 43 is used to input the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar. The avatar style generation module is pre-trained.

[0141] The detection module 44 is used to detect the facial key points of the cartoon avatar and the predicted cartoon avatar, and to determine the facial key point loss;

[0142] The determination module 45 is used to determine the image conversion loss based on the difference between the cartoon avatar and the predicted cartoon avatar;

[0143] The update module 46 is used to update the parameters of the feature encoding module based on the fusion result of the facial key point loss and the image transformation loss, so as to determine the target avatar generation model.

[0144] In some implementations, the update module 46 includes:

[0145] The first acquisition unit is used to acquire multiple sets of weight combinations, wherein the weight combinations include weights that correspond one-to-one with the facial key point loss and the image transformation loss.

[0146] The fusion unit is used to fuse the facial key point loss and the image transformation loss for each of the weight combinations to obtain the corresponding comprehensive loss;

[0147] An update unit is used to update the parameters of the feature encoding module using the comprehensive loss to obtain an optional avatar generation model;

[0148] The first determining unit is configured to determine the target avatar generation model from the optional avatar generation models based on the training effect of the optional avatar generation models, wherein the training effect includes at least one of training time and accuracy of the optional avatar generation models.

[0149] In some implementations, the first acquisition unit includes:

[0150] Obtain sub-units to obtain the value range of each of the weights;

[0151] A sub-unit is generated to generate the values ​​of the weights based on the value range, so as to obtain the multiple sets of weight combinations.

[0152] In some implementations, the determining module 45 includes:

[0153] An extraction unit is used to extract facial features from the cartoon avatar and the predicted cartoon avatar respectively, to obtain a first facial feature and a second facial feature;

[0154] The first calculation unit is used to perform loss calculation based on the first face feature and the second face feature to determine the feature loss;

[0155] The second calculation unit is used to calculate pixel loss and perceptual loss based on the cartoon avatar and the predicted cartoon avatar, respectively. The image conversion loss includes the feature loss, the pixel loss and the perceptual loss.

[0156] In some implementations, the feature encoding module is constructed based on a feature pyramid network, and the first input module 42 includes:

[0157] The first input unit is used to input the real headshot into the feature encoding module and extract multiple pyramid-scale feature vectors to determine the feature encoding.

[0158] In some implementations, the first input module 43 includes:

[0159] The second input unit is used to input the feature vectors into the corresponding network layer of the avatar style generation module for image reconstruction based on the pyramid scale of each feature vector in the feature encoding, so as to determine the predicted cartoon avatar.

[0160] This embodiment also provides a device for generating comic book-style avatars, such as... Figure 6 As shown, it includes:

[0161] The second acquisition module 51 is used to acquire the face image to be processed;

[0162] The third input module 52 is used to input the face image to be processed into the target avatar generation model to obtain the target cartoon avatar. The target avatar generation model is trained according to the training method of any of the above-described American comic style avatar generation models.

[0163] In some implementations, the second acquisition module 51 includes:

[0164] The second acquisition unit is used to acquire the image to be processed, which contains a human face;

[0165] The detection unit is used to perform face detection on the image to be processed to obtain a face image;

[0166] The second determining unit is used to perform face alignment and normalization processing based on the face image to determine the face image to be processed.

[0167] In this embodiment, the training device for generating comic book-style avatars and the device for generating comic book-style avatars are presented in the form of functional units. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0168] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0169] This invention also provides an electronic device having the above-described features. Figure 5 The training device shown is for the American comic-style avatar generation model, or Figure 6 The illustrated avatar generator features an American comic book style.

[0170] Please see Figure 7 , Figure 7This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include: at least one processor 61, such as a CPU (Central Processing Unit), at least one communication interface 63, memory 64, and at least one communication bus 62. The communication bus 62 is used to enable communication between these components. The communication interface 63 may include a display screen or a keyboard; optionally, the communication interface 63 may also include a standard wired interface or a wireless interface. The memory 64 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 64 may also be at least one storage device located remotely from the aforementioned processor 61. The processor 61 may be combined with... Figure 5 or Figure 6 The described apparatus has an application program stored in memory 64, and the processor 61 calls the program code stored in memory 64 to perform any of the above method steps.

[0171] The communication bus 62 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0172] The memory 64 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 64 may also include a combination of the above types of memory.

[0173] The processor 61 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0174] The processor 61 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0175] Optionally, memory 64 is also used to store program instructions. Processor 61 can call program instructions to implement the training method for the comic book-style avatar generation model, or the comic book-style avatar generation method, as shown in any embodiment of this application.

[0176] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the training method for the comic book-style avatar generation model, or the comic book-style avatar generation method, in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0177] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A training method for a comic book-style avatar generation model, characterized in that, include: Obtain sample image pairs, which include real headshots and corresponding cartoon headshots; The real headshot is input into the feature encoding module of the headshot generation model to determine the feature encoding; The feature encoding is input into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar. The avatar style generation module is pre-trained. Detect the facial landmarks of the cartoon avatar and the predicted cartoon avatar, and determine the facial landmark loss; Based on the difference between the cartoon avatar and the predicted cartoon avatar, the image conversion loss is determined; Based on the fusion result of the facial key point loss and the image transformation loss, the parameters of the feature encoding module are updated to determine the target avatar generation model; The step of updating the parameters of the feature encoding module based on the fusion result of the facial key point loss and the image transformation loss to determine the target avatar generation model includes: Obtain multiple sets of weight combinations, wherein the weight combinations include weights that correspond one-to-one with the facial key point loss and the image transformation loss; For each of the weight combinations, the facial key point loss and the image transformation loss are fused to obtain the corresponding comprehensive loss; The parameters of the feature encoding module are updated using the comprehensive loss to obtain an optional avatar generation model; Based on the training performance of the optional avatar generation models, the target avatar generation model is determined from the optional avatar generation models, wherein the training performance includes at least one of training time and accuracy of the optional avatar generation models.

2. The training method according to claim 1, characterized in that, The process of obtaining multiple weight combinations includes: Obtain the value range of each of the weights; The values ​​of the weights are generated based on the range of values ​​to obtain the multiple sets of weight combinations.

3. The training method according to claim 1, characterized in that, The step of determining the image transformation loss based on the difference between the cartoon avatar and the predicted cartoon avatar includes: Facial features are extracted from the cartoon avatar and the predicted cartoon avatar respectively to obtain the first facial feature and the second facial feature; Loss is calculated based on the first facial feature and the second facial feature to determine the feature loss; Based on the cartoon avatar and the predicted cartoon avatar, pixel loss and perceptual loss are calculated respectively. The image transformation loss includes the feature loss, the pixel loss and the perceptual loss.

4. The training method according to claim 1, characterized in that, The feature encoding module is built based on a feature pyramid network. The step of inputting the real avatar into the feature encoding module of the avatar generation model and determining the feature encoding includes: The real headshot is input into the feature encoding module, and multiple pyramid-scale feature vectors are extracted to determine the feature encoding.

5. The training method according to claim 4, characterized in that, The step of inputting the feature encoding into the avatar style generation module of the avatar generation model to determine the predicted cartoon avatar includes: Based on the pyramid scale of each feature vector in the feature encoding, the feature vector is input into the corresponding network layer of the avatar style generation module for image reconstruction to determine the predicted cartoon avatar.

6. A method for generating comic book-style avatars, characterized in that, include: Obtain the face image to be processed; The face image to be processed is input into the target avatar generation model to obtain the target cartoon avatar. The target avatar generation model is trained by the training method of the American comic style avatar generation model according to any one of claims 1-5.

7. The method according to claim 6, characterized in that, The process of acquiring the face image to be processed includes: Obtain the image to be processed, which contains human faces; Face detection is performed on the image to be processed to obtain a face image; Based on the face image, face alignment and normalization processing are performed to determine the face image to be processed.

8. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the training method for the comic book-style avatar generation model according to any one of claims 1-5, or to perform the comic book-style avatar generation method according to claim 6 or 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the training method of the comic book-style avatar generation model according to any one of claims 1-5, or to perform the comic book-style avatar generation method according to claim 6 or 7.

Images