Facial image processing method and device, model training method and device and electronic equipment

By extracting and transforming the reference facial image and the original facial image, the problems of low facial processing efficiency and poor beautification effect in the prior art are solved, and facial image processing with efficient beautification effect is achieved.

CN118941439APending Publication Date: 2024-11-12BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410973061.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When performing facial image processing, it is difficult for the prior art to maintain the beautification effect of the result image while ensuring processing efficiency. The existing makeup migration method will cause all makeup of the user's facial image to be replaced, and the beautification effect is poor.

Method used

By performing feature extraction and feature transformation on the reference facial image and the original facial image, image features are updated to preserve the facial features of the original facial image while maintaining processing efficiency and improving the beautification effect.

Benefits of technology

It realizes that while improving the efficiency of facial image processing, maintaining the facial features of the result image, improving the beautification effect, and ensuring that the result image retains some of the features of the original facial image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941439A_ABST
    Figure CN118941439A_ABST
Patent Text Reader

Abstract

The invention provides a facial image processing method, a model training method and device and electronic equipment, and belongs to the technical field of computers. According to the method, feature transformation is performed on image features of at least one reference face image based on image features of an original face image, so that the editing degree of the image features of the at least one reference face image on the original face image can be updated; when the image features of the original face image and the third image features obtained through feature transformation are decoded, a result image obtained through decoding can keep part of the image features of the original face image, namely, the result image can have the face features of the original face image; therefore, the beautifying effect of the processed face image can be improved while the processing efficiency of the face image is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a facial image processing method, a model training method, a device and an electronic device. Background Art

[0002] With the development of Internet technology and artificial intelligence technology, users can use various virtual tools in applications to process facial images to beautify them. Facial processing includes a variety of processing methods, such as whitening, skin resurfacing, makeup, and face slimming. However, during the facial processing process, users are required to perform multiple manual operations on facial images, which is a large workload and leads to low image processing efficiency. Therefore, how to improve the image processing efficiency when performing facial processing on facial images is a technical problem that needs to be solved.

[0003] In the related art, taking a facial processing method as an example, a makeup transfer method can be used to process a facial image. This method can transfer the makeup of a person in a reference facial image to a facial image provided by a user through a makeup transfer model, so that a processed facial image can be quickly obtained.

[0004] However, when performing facial processing, the above solution will replace all the makeup of the facial image provided by the user with the makeup of the person in the reference image, resulting in a poor beautification effect of the processed facial image. Summary of the invention

[0005] The present disclosure provides a facial image processing method, a model training method, a device and an electronic device, so that the result image can have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image while improving the beautification effect of the processed facial image. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, a facial image processing method is provided, including:

[0007] Processing at least one reference facial image to obtain a first image feature;

[0008] Extracting features from the original facial image to obtain second image features;

[0009] Based on the second image feature, performing a feature transformation on the first image feature to obtain a third image feature, wherein the feature transformation is used to update the editing degree of the first image feature on the original facial image;

[0010] The second image feature and the third image feature are decoded to obtain a result image.

[0011] According to one aspect of an embodiment of the present disclosure, a model training method is provided, including:

[0012] Processing at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image;

[0013] Determining a first training loss based on the at least one sample reference facial image and the prediction result image, the first training loss being used to indicate a difference between the at least one sample reference facial image and the prediction result image;

[0014] Determining a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;

[0015] The image processing model is trained based on the first training loss and the second training loss.

[0016] According to another aspect of an embodiment of the present disclosure, there is provided a facial image processing apparatus, comprising:

[0017] a processing unit configured to process at least one reference facial image to obtain a first image feature;

[0018] A feature extraction unit is configured to extract features from the original facial image to obtain second image features;

[0019] a feature transformation unit configured to perform feature transformation on the first image feature based on the second image feature to obtain a third image feature, wherein the feature transformation is used to update the editing degree of the first image feature on the original facial image;

[0020] A decoding unit is configured to decode the second image feature and the third image feature to obtain a result image.

[0021] In some embodiments, the processing unit includes:

[0022] an extraction subunit, configured to perform feature extraction on the at least one reference facial image respectively to obtain image features of the at least one reference facial image;

[0023] The fusion subunit is configured to perform weighted fusion on the image features of the at least one reference facial image based on the feature fusion weight corresponding to the at least one reference facial image to obtain the first image feature.

[0024] In some embodiments, the processing unit further includes:

[0025] The first determination subunit is configured to determine, for any reference facial image, a facial angle and a facial clarity of the reference facial image based on image features of the reference facial image, wherein the facial angle is used to indicate a degree of deflection of the face in the reference facial image, and the facial clarity is used to indicate a degree of clarity of the face in the reference facial image; and determine a feature fusion weight corresponding to the reference facial image based on at least one of the facial angle and the facial clarity.

[0026] In some embodiments, the feature conversion unit includes:

[0027] A second determining subunit is configured to determine, based on first sub-feature pairs, similarities corresponding to the first sub-feature pairs, each first sub-feature pair comprising sub-features at corresponding positions in the first image feature and the second image feature, the similarity being used to indicate a degree of similarity between two sub-features in the first sub-feature pair;

[0028] The transformation subunit is configured to perform feature transformation on the first image feature based on the similarities corresponding to the plurality of first sub-feature pairs to obtain the third image feature.

[0029] In some embodiments, the first sub-feature pair includes a first sub-feature from the first image feature and a second sub-feature from the second image feature;

[0030] The transformation subunit is configured to update, for any first sub-feature pair, a feature transformation weight of the first sub-feature based on the similarity corresponding to the first sub-feature pair, wherein the feature transformation weight is used to indicate the degree of editing of the original facial image by the first sub-feature; and weight the first sub-feature based on the feature transformation weight to obtain a third sub-feature, wherein the third sub-feature is a sub-feature at a corresponding position in the third image feature.

[0031] In some embodiments, the transformation subunit is configured to reduce the feature transformation weight when the similarity corresponding to the first sub-feature pair is greater than a first threshold; and increase the feature transformation weight when the similarity corresponding to the first sub-feature pair is less than a second threshold, and the second threshold is not greater than the first threshold.

[0032] In some embodiments, the decoding unit is configured to determine the editing degree corresponding to the second sub-feature pair based on the second sub-feature pair, each second sub-feature pair includes the sub-features at corresponding positions in the second image feature and the third image feature, and the editing degree is used to indicate the editing degree of the third sub-feature in the second sub-feature pair to the second sub-feature, the third sub-feature is the sub-feature at the corresponding position in the third image feature, and the second sub-feature is the sub-feature at the corresponding position in the second image feature, and the editing degree is proportional to the difference between the second sub-feature and the third sub-feature; based on the editing degrees corresponding to the multiple second sub-feature pairs, the second image feature is decoded to obtain the result image.

[0033] According to another aspect of an embodiment of the present disclosure, a model training device is provided, including:

[0034] A processing unit is configured to process at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image;

[0035] A first determining unit is configured to determine a first training loss based on the at least one sample reference facial image and the prediction result image, wherein the first training loss is used to indicate a difference between the at least one sample reference facial image and the prediction result image;

[0036] A second determining unit is configured to determine a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;

[0037] A training unit is configured to train the image processing model based on the first training loss and the second training loss.

[0038] In some embodiments, the processing unit is configured to process the at least one sample reference facial image through the image processing model to obtain a first sample image feature; to extract features from the sample original image through the image processing model to obtain a second sample image feature; to perform feature transformation on the first sample image feature based on the second sample image feature through the image processing model to obtain a third sample image feature, wherein the feature transformation is used to update the degree of editing of the sample original facial image by the first sample image feature; and to decode the second sample image feature and the third sample image feature through the image processing model to obtain the predicted result image.

[0039] In some embodiments, the processing unit is configured to determine, through the image processing model, the similarities corresponding to the sample sub-feature pairs based on the sample sub-feature pairs in the first sample image feature and the sample second image feature, each sample sub-feature pair includes sample sub-features at corresponding positions in the first sample image feature and the second sample image feature, and the similarity is used to indicate the degree of similarity between the two sample sub-features in the sample sub-feature pair; based on the similarities corresponding to multiple sample sub-feature pairs, perform feature transformation on the first sample image feature to obtain the third sample image feature.

[0040] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device comprising:

[0041] one or more processors;

[0042] A memory for storing program codes executable by the processor;

[0043] Among them, the processor is configured to execute the program code to implement the above-mentioned facial image processing method or the above-mentioned model training method.

[0044] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned facial image processing method or the above-mentioned model training method.

[0045] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned facial image processing method or the above-mentioned model training method when executed by a processor.

[0046] The disclosed embodiment provides a facial image processing method, which can update the editing degree of the image features of the at least one reference facial image to the original facial image by performing feature transformation on the image features of the original facial image based on the image features of the original facial image, and then when decoding the image features of the original facial image and the third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image while improving the beautification effect of the processed facial image.

[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0049] Figure 1 The figure is a schematic diagram of an implementation environment of a facial image processing method according to an exemplary embodiment.

[0050] Figure 2 The figure is a flowchart of a facial image processing method according to an exemplary embodiment.

[0051] Figure 3 It is a structural schematic diagram of an image processing model according to an exemplary embodiment.

[0052] Figure 4 The figure is a flowchart of another facial image processing method according to an exemplary embodiment.

[0053] Figure 5 The invention is a block diagram of a facial image processing apparatus according to an exemplary embodiment.

[0054] Figure 6 It is a block diagram of a model training device according to an exemplary embodiment.

[0055] Figure 7 It is a block diagram of a terminal according to an exemplary embodiment.

[0056] Figure 8 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0057] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0058] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0059] The user information involved in this disclosure may be information authorized by the user or fully authorized by all parties. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the facial images involved in this application are all obtained with full authorization.

[0060] The facial image processing method provided by the embodiments of the present disclosure can be performed by an electronic device. In some embodiments, the electronic device can be provided as a terminal or a server. When the electronic device is provided as a terminal, the terminal can implement the operations performed by the facial image processing method; when it is provided as a server, the server can implement the operations performed by the facial image processing method. The server can receive a facial image authorized by the terminal, and the server processes the facial image based on the image processing model; the server and the terminal can also interact to implement the operations performed by the facial image processing method; the terminal can also send an image processing request to the server, and the server performs image processing, and then feeds back the result of the image processing to the terminal, and the terminal outputs the result of the image processing.

[0061] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a facial image processing method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.

[0062] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and a user may log in to the application through the terminal 101 to obtain the services provided by the application. The server 102 may be a background server of the application, and is used to provide background services for the application. The terminal 101 may be connected to the server 102 via a wireless network or a wired network, and the facial image to be processed may be sent to the server 102.

[0063] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be only a few, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0064] The server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. The server 102 can receive the facial image sent by the terminal 101, and return the result image obtained by processing the facial image to the terminal 101 for the user to view. Optionally, the number of the above servers can be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0065] Figure 2 is a flowchart of a facial image processing method according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0066] In step S201, the electronic device processes at least one reference facial image to obtain a first image feature.

[0067] In the embodiment of the present disclosure, the reference facial image is an image including the face of any object. The object may be a person or an animal, which is not limited in the embodiment of the present disclosure. The electronic device can process at least one reference facial image to obtain a first image feature for representing at least one reference facial image. The number of at least one reference facial image may be one or more, which is not limited in the embodiment of the present disclosure.

[0068] In step S202, the electronic device extracts features from the original facial image to obtain second image features.

[0069] In the disclosed embodiment, the original facial image may be a facial image of the user himself, or a facial image taken by the user for another object, or a facial image of any object downloaded by the user from the network, and the disclosed embodiment does not limit this. The object included in the original facial image may be the same as or different from the object included in the reference facial image, and the disclosed embodiment does not limit this. The electronic device can extract features from the original facial image to obtain second image features for representing the original facial image.

[0070] In step S203, the electronic device performs feature transformation on the first image feature based on the second image feature to obtain a third image feature, and the feature transformation is used to update the editing degree of the first image feature on the original facial image.

[0071] In the disclosed embodiment, since the electronic device needs to edit the original facial image based on at least one reference facial image to obtain the beautified original facial image, the electronic device can edit the second image feature based on the first image feature to obtain the image feature of the beautified original facial image. However, if the image features of the original facial image are all replaced with the image features of at least one reference facial image during the editing process, the beautified original facial image will lack the facial features of the original facial image. In order to solve this problem, the electronic device can perform feature transformation on the first image feature based on the second image feature to update the editing degree of the first image feature to the original facial image. The third image feature is the first image feature after the feature transformation.

[0072] In step S204, the electronic device decodes the second image feature and the third image feature to obtain a result image.

[0073] In the disclosed embodiment, the result image is an image obtained by editing the original facial image based on at least one reference facial image. After the first image feature is transformed to obtain the third image feature, the electronic device can decode the second image feature and the third image feature to obtain the result image. The result image includes partial image features of at least one reference facial image and retains partial image features of the original facial image.

[0074] The present disclosure provides a facial image processing method, which can update the editing degree of the image features of the at least one reference facial image to the original facial image by performing feature transformation on the image features of the original facial image based on the image features of the original facial image, and then when decoding the image features of the original facial image and the third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, so that the beautification effect of the processed facial image can be improved while ensuring the processing efficiency of the facial image.

[0075] In some embodiments, processing at least one reference facial image to obtain a first image feature includes:

[0076] Extracting features from at least one reference facial image respectively to obtain image features of at least one reference facial image;

[0077] Based on the feature fusion weight corresponding to the at least one reference facial image, the image features of the at least one reference facial image are weightedly fused to obtain a first image feature.

[0078] In the disclosed embodiment, by weighted fusion of the image features of at least one reference facial image based on the feature fusion weight calculated by the server, the feature extraction module can be focused on the more valuable reference facial image, thereby improving the quality of the fused first image features. By weighted fusion of the image features of at least one reference facial image based on the feature fusion weight manually set by the user, the feature extraction module can be focused on the reference facial image with a larger weight set by the user, thereby making the fused first image features more in line with user needs.

[0079] In some embodiments, the method further comprises:

[0080] For any reference facial image, determining a facial angle and facial clarity of the reference facial image based on image features of the reference facial image, wherein the facial angle is used to indicate a degree of deflection of the face in the reference facial image, and the facial clarity is used to indicate a degree of clarity of the face in the reference facial image;

[0081] Based on at least one of the facial angle and the facial definition, a feature fusion weight corresponding to the reference facial image is determined.

[0082] In the embodiment of the present disclosure, by determining the feature fusion weight corresponding to the reference facial image based on at least one of the facial angle and facial clarity of the reference facial image, the feature fusion weight corresponding to the reference facial image with higher facial quality can be made larger, so that the fused first image features can be concentrated on the reference facial image with higher facial quality, thereby improving the quality of the first image features.

[0083] In some embodiments, based on the second image feature, performing feature transformation on the first image feature to obtain the third image feature includes:

[0084] Based on the first sub-feature pairs, determining the similarity corresponding to the first sub-feature pairs, each first sub-feature pair includes sub-features at corresponding positions in the first image feature and the second image feature, and the similarity is used to indicate the degree of similarity between the two sub-features in the first sub-feature pair;

[0085] Based on the similarities corresponding to the plurality of first sub-feature pairs, feature transformation is performed on the first image feature to obtain a third image feature.

[0086] In the disclosed embodiment, by performing feature transformation on the first image feature based on the corresponding similarity of the first sub-feature pair, the feature transformation module can update the editing degree of the sub-features at corresponding positions in the first image feature for the original facial image based on the similarity between the sub-features at corresponding positions in the first image feature and the second image feature, so that the result image obtained based on the image features of the original facial image and the third image features obtained by the feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.

[0087] In some embodiments, the first sub-feature pair includes a first sub-feature from the first image feature and a second sub-feature from the second image feature;

[0088] Based on the similarities corresponding to the plurality of first sub-feature pairs, the first image feature is subjected to feature transformation to obtain a third image feature, including:

[0089] For any first sub-feature pair, based on the similarity corresponding to the first sub-feature pair, updating the feature transformation weight of the first sub-feature, the feature transformation weight is used to indicate the editing degree of the first sub-feature to the original facial image;

[0090] Based on the feature transformation weight, the first sub-feature is weighted to obtain a third sub-feature, where the third sub-feature is a sub-feature at a corresponding position in the third image feature.

[0091] In the embodiment of the present disclosure, by updating the feature transformation weight of the first sub-feature based on the corresponding similarity of the first sub-feature, the editing degree of the original facial image by the first sub-feature can be changed as the similarity between the first sub-feature and the second sub-feature changes, so that the result image obtained based on the image features of the original facial image and the third image features obtained by the feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.

[0092] In some embodiments, updating the feature transformation weight of the first sub-feature based on the corresponding similarity of the first sub-feature pair includes:

[0093] When the similarity corresponding to the first sub-feature pair is greater than a first threshold, reducing the feature transformation weight;

[0094] When the similarity corresponding to the first sub-feature pair is less than the second threshold, the feature transformation weight is increased, and the second threshold is not greater than the first threshold.

[0095] In the embodiment of the present disclosure, by reducing the feature transformation weight of the first sub-feature when the similarity between the first sub-feature and the second sub-feature is high, and increasing the feature change weight of the first sub-feature when the similarity between the first sub-feature and the second sub-feature is low, the result image obtained based on the second image feature and the third image feature obtained by feature transformation can retain only some image features in the original facial image that have a high similarity with the first image feature, that is, the result image can retain only the facial features with higher quality in the original facial image, thereby improving the beautification effect of the result image.

[0096] In some embodiments, decoding the second image feature and the third image feature to obtain a result image includes:

[0097] Based on the second sub-feature pairs, determining the editing degree corresponding to the second sub-feature pairs, each second sub-feature pair includes sub-features at corresponding positions in the second image feature and the third image feature, the editing degree is used to indicate the editing degree of the second sub-feature by the third sub-feature in the second sub-feature pair, the third sub-feature is the sub-feature at the corresponding position in the third image feature, the second sub-feature is the sub-feature at the corresponding position in the second image feature, and the editing degree is proportional to the difference between the second sub-feature and the third sub-feature;

[0098] Based on the editing degrees corresponding to the plurality of second sub-feature pairs, the second image feature is decoded to obtain a result image.

[0099] In the disclosed embodiment, by decoding the second image feature based on the corresponding editing degree of multiple second sub-features, the original facial image can be edited according to the editing degree of the original facial image of the third image feature obtained by feature transformation, so that the edited result image can retain some image features of the original facial image, that is, the result image can have the facial characteristics of the original facial image, thereby improving the beautification effect of the result image.

[0100] In some embodiments, the training process of the image processing model includes:

[0101] Processing at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image;

[0102] Determining a first training loss based on at least one sample reference facial image and the prediction result image, the first training loss being used to indicate a difference between the at least one sample reference facial image and the prediction result image;

[0103] Based on the sample original facial image and the predicted result image, determining a second training loss, the second training loss being used to indicate a difference between the sample original facial image and the predicted result image;

[0104] The image processing model is trained based on the first training loss and the second training loss.

[0105] In the disclosed embodiment, by training the image processing model based on the first training loss and the second training loss, the facial features in the result image output by the trained image processing model can be made consistent with the reference facial image after feature transformation, and other image features can be made consistent with the original facial image, so that the result image not only includes the image features of the reference facial image, but also retains the image features of the original facial image, thereby improving the accuracy of the image processing model.

[0106] In some embodiments, at least one sample reference facial image and a sample original facial image are processed by an image processing model to obtain a prediction result image, including:

[0107] Processing at least one sample reference facial image through an image processing model to obtain a first sample image feature;

[0108] Through the image processing model, feature extraction is performed on the sample original image to obtain the second sample image feature;

[0109] Through the image processing model, based on the second sample image feature, the first sample image feature is subjected to feature transformation to obtain a third sample image feature, wherein the feature transformation is used to update the editing degree of the first sample image feature to the sample original facial image;

[0110] The second sample image features and the third sample image features are decoded through the image processing model to obtain a prediction result image.

[0111] In the embodiment of the present disclosure, by performing feature transformation on the first sample image feature based on the second sample image feature, the editing degree of the sample original facial image by the first sample image feature can be updated, and then when the image features of the sample original facial image and the third sample image features obtained by the feature transformation are decoded, the decoded prediction result image can retain some image features of the sample original facial image, that is, the prediction result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the prediction result image output by the image processing model.

[0112] In some embodiments, performing feature transformation on the first sample image feature based on the second sample image feature through the image processing model to obtain the third sample image feature includes:

[0113] Determine, by an image processing model, the similarity corresponding to the sample sub-feature pairs based on the sample sub-feature pairs in the first sample image feature and the sample second image feature, each sample sub-feature pair includes sample sub-features at corresponding positions in the first sample image feature and the second sample image feature, and the similarity is used to indicate the degree of similarity between the two sample sub-features in the sample sub-feature pair;

[0114] Based on the similarities corresponding to the plurality of sample sub-feature pairs, feature transformation is performed on the first sample image feature to obtain a third sample image feature.

[0115] In the disclosed embodiment, by performing feature transformation on the first sample image feature based on the corresponding similarities of multiple sample sub-feature pairs, the feature transformation module can update the editing degree of the sample original facial image by the sample sub-features at corresponding positions in the first sample image feature based on the similarity between the sub-features at corresponding positions in the first image feature and the second image feature, so that the predicted result image obtained based on the image features of the sample original facial image and the third image features obtained by the feature transformation can selectively retain some image features in the sample original facial image, that is, the predicted result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the predicted result image.

[0116] Based on the above embodiments, the image processing method can be implemented based on an image processing model. For example, Figure 3 FIG. 1 is a schematic diagram showing a structure of an image processing model according to an exemplary embodiment. Figure 3 As shown, the image processing model includes a feature extraction module, a feature transformation module and an image editing module. Among them, the feature extraction module is used to process the input image. The feature transformation module is used to perform feature transformation on the image features. The encoder in the image editing module is used to extract features from the input image. The decoder in the image editing module is used to decode the image features to reconstruct the image. Specifically, when multiple reference facial images and an original facial image are input into the image processing model, the feature extractor in the feature extraction module can extract features from the multiple reference facial images to obtain the image features of the multiple reference facial images, that is, the corresponding Figure 3The feature 1, feature 2 and feature n shown. Then the feature extraction module can also perform weighted fusion on the image features of multiple reference facial images based on the labels corresponding to the multiple reference facial images to obtain the first image feature. The label is used to represent the feature fusion weight of the reference facial image. The encoder in the image editing module can perform feature extraction on the original facial image to obtain the second image feature. After obtaining the first image feature and the second image feature, the feature transformation module can perform feature transformation on the first image feature based on the second image feature to obtain the third image feature. Finally, the decoder in the image editing module can decode the second image feature and the third image feature to obtain the result image.

[0117] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 4 FIG. 1 is a flowchart of another facial image processing method according to an exemplary embodiment. Figure 4 , the method comprising:

[0118] In step S401, the server extracts features from a plurality of reference facial images respectively through a feature extraction module in an image processing model to obtain image features of the plurality of reference facial images.

[0119] In the disclosed embodiment, the reference facial image may be a facial image uploaded by a user, or a facial image selected by a user from a facial image database, and the disclosed embodiment does not limit this. After acquiring multiple reference facial images, the server can first extract features from the multiple reference facial images respectively through the feature extraction module in the image processing model to obtain image features of each reference facial image. The image features of each reference facial image include multiple facial features of the object in the reference facial image, such as skin color and texture features, facial features, and makeup features.

[0120] Optionally, the feature extraction module of the image processing model can extract features of the reference facial image from a specific dimension. Taking the specific dimension as the skin color dimension as an example, the feature extraction module can extract facial skin color features from the reference facial image. Taking the specific dimension as the makeup dimension as an example, the feature extraction module can extract facial makeup features from the reference facial image. By extracting features of the reference facial image from a specific dimension, a variety of facial features contained in the reference facial image can be more accurately extracted.

[0121] In step S402, the server performs weighted fusion on the image features of the multiple reference facial images based on the feature fusion weights corresponding to the multiple reference facial images through the feature extraction module in the image processing model to obtain a first image feature.

[0122] In the embodiment of the present disclosure, each reference facial image corresponds to a feature fusion weight, which is used to weight the image features of the reference facial image. The feature fusion weight can be manually set by the user, or can be automatically calculated by the server based on the image features of at least one reference facial image, which is not limited in the embodiment of the present disclosure. Accordingly, after obtaining the image features of at least one reference facial image through the feature extraction module in the image processing model, the server can continue to perform weighted fusion on the image features of at least one reference facial image through the feature extraction module based on the feature fusion weight corresponding to at least one reference facial image to obtain the first image feature. The first image feature is used to represent at least one reference facial image. By weighted fusion of the image features of at least one reference facial image based on the feature fusion weight calculated by the server, the feature extraction module can be focused on more valuable reference facial images, thereby improving the quality of the fused first image features. By weighted fusion of the image features of at least one reference facial image based on the feature fusion weight manually set by the user, the feature extraction module can be focused on the reference facial image with a larger weight set by the user, thereby making the fused first image features more in line with user needs.

[0123] It should be noted that the above steps 401-402 are described by taking the case where there are multiple reference facial images as an example. When there is only one reference facial image, the server extracts features from the reference facial image through a feature extraction module to obtain the first image feature.

[0124] In some embodiments, the server can calculate the feature fusion weight corresponding to the reference facial image based on the image features of the reference facial image. Accordingly, for any reference facial image, the server can determine the facial angle and facial clarity of the reference facial image based on the image features of the reference facial image. Since the facial angle is used to indicate the degree of deflection of the face in the reference facial image, and the facial clarity is used to indicate the clarity of the face in the reference facial image. Therefore, the server can determine the facial quality of the reference facial image based on at least one of the facial angle and the facial clarity, and then can determine the feature fusion weight corresponding to the reference facial image according to the facial quality. Among them, the smaller the facial angle of the reference facial image, the smaller the degree of deflection of the face in the reference facial image, that is, the more positive the face in the reference facial image, and therefore the higher the facial quality of the reference facial image. The higher the facial clarity of the reference facial image, the higher the clarity of the face in the reference facial image, that is, the clearer the face in the reference facial image, and therefore the higher the facial quality of the reference facial image. In the case where the facial clarity of at least one reference facial image is the same, the feature fusion weight corresponding to the reference facial image is negatively correlated with the facial angle of the reference facial image, that is, the smaller the facial angle of the reference facial image, the larger the feature fusion weight corresponding to the reference facial image. In the case where the facial angle of at least one reference facial image is the same, the feature fusion weight corresponding to the reference facial image is positively correlated with the facial clarity of the reference facial image, that is, the higher the facial clarity of the reference facial image, the larger the feature fusion weight corresponding to the reference facial image. By determining the feature fusion weight corresponding to the reference facial image based on at least one of the facial angle and facial clarity of the reference facial image, the feature fusion weight corresponding to the reference facial image with higher facial quality can be made larger, so that the fused first image features can be concentrated on the reference facial image with higher facial quality, and the quality of the first image features can be improved.

[0125] In step S403, the server extracts features from the original facial image through the image editing module in the image processing model to obtain second image features.

[0126] In the disclosed embodiment, the image editing module includes an encoder for extracting features from the image. Accordingly, the server can extract features from the original facial image through the image editing module in the image processing model to obtain second image features. The second image features include multiple facial features of the object in the original facial image, such as skin color and texture features, facial features, makeup features, etc. The original facial image can be the user's own facial image, or a facial image taken by the user for other objects, or a facial image of any object downloaded by the user from the network, and the disclosed embodiment does not limit this.

[0127] In step S404, the server determines the similarity corresponding to the first sub-feature pairs based on the first sub-feature pairs through the feature transformation module in the image processing model, each first sub-feature pair includes sub-features at corresponding positions in the first image feature and the second image feature, and the similarity is used to indicate the degree of similarity between the two sub-features in the first sub-feature pair.

[0128] In the disclosed embodiment, the first image feature and the second image feature both include multiple element positions, and each element position has a corresponding sub-feature. Therefore, the server can determine multiple first sub-feature pairs based on the first image feature and the second image feature. Each first sub-feature pair includes sub-features at corresponding positions in the first image feature and the second image feature. Optionally, the multiple sub-features are multiple facial features of an object in a facial image, such as skin color and texture features, facial features, and makeup features.

[0129] In the process of performing feature transformation on the first image feature based on the second image feature, the server can first determine the similarity between the first image feature and the second image feature, and then perform feature transformation on the first image feature based on the similarity. Accordingly, when determining the similarity, the server can determine the similarity corresponding to each first sub-feature pair. That is, for any first sub-feature pair, the server can determine the similarity corresponding to the first sub-feature pair based on the two sub-features in the first sub-feature pair through the feature transformation module in the image processing model. Among them, the similarity corresponding to the first sub-feature pair is used to indicate the degree of similarity between the two sub-features in the first sub-feature pair.

[0130] In step S405, the server performs feature transformation on the first image feature based on the similarities corresponding to multiple first sub-feature pairs through the feature transformation module in the image processing model to obtain a third image feature. The feature transformation is used to update the editing degree of the first image feature to the original facial image.

[0131] In the disclosed embodiment, after determining the similarities corresponding to the plurality of first sub-feature pairs, the server can continue to perform feature transformation on the first image feature based on the similarities corresponding to the plurality of first sub-feature pairs through the feature transformation module to update the editing degree of the first image feature to the original facial image. The third image feature is the first image feature after feature transformation. By performing feature transformation on the first image feature based on the similarities corresponding to the first sub-feature pairs, the feature transformation module can update the editing degree of the sub-features at the corresponding positions in the first image feature to the original facial image based on the similarities between the sub-features at the corresponding positions in the first image feature and the second image feature, so that the result image obtained based on the image features of the original facial image and the third image feature obtained by feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.

[0132] In some embodiments, for any first sub-feature pair, the server can perform feature transformation on the sub-feature at the corresponding position in the first image feature based on the similarity corresponding to the first sub-feature pair. Since the first sub-feature pair includes a first sub-feature from the first image feature and a second sub-feature from the second image feature, that is, the first sub-feature is a sub-feature at the corresponding position in the first image feature, and the second sub-feature is a sub-feature at the corresponding position in the second image feature. Therefore, in the process of performing feature transformation on the sub-feature at the corresponding position in the first image feature, the server can first update the feature transformation weight of the first sub-feature based on the similarity corresponding to the first sub-feature pair. Since the feature transformation weight is used to indicate the degree of editing of the original facial image by the first sub-feature, the server can weight the first sub-feature based on the feature transformation weight to obtain the first sub-feature after feature transformation, that is, the third sub-feature. Among them, the third sub-feature is the sub-feature at the corresponding position in the third image feature obtained by performing feature transformation on the first image feature. By updating the feature transformation weight of the first sub-feature based on the corresponding similarity of the first sub-feature, the editing degree of the original facial image by the first sub-feature can be changed as the similarity between the first sub-feature and the second sub-feature changes, so that the result image obtained based on the image features of the original facial image and the third image features obtained by feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.

[0133] In some embodiments, the server can increase or decrease the feature transformation weight of the first sub-feature based on the level of similarity corresponding to the first sub-feature pair. Accordingly, when the similarity corresponding to the first sub-feature pair is greater than the first threshold, it indicates that the similarity between the first sub-feature and the second sub-feature is high, that is, it indicates that the second sub-feature is a facial feature of higher quality in the original facial image. Therefore, the server can reduce the feature transformation weight of the first sub-feature, that is, reduce the degree of editing of the original facial image by the first sub-feature; when the similarity corresponding to the first sub-feature pair is less than the second threshold, it indicates that the similarity between the first sub-feature and the second sub-feature is low, that is, it indicates that the second sub-feature is a facial feature of lower quality in the original facial image. Therefore, the server can increase the feature transformation weight of the first sub-feature, that is, increase the degree of editing of the original facial image by the first sub-feature. Wherein, the second threshold is not greater than the first threshold. By reducing the feature transformation weight of the first sub-feature when the similarity between the first sub-feature and the second sub-feature is high, and increasing the feature change weight of the first sub-feature when the similarity between the first sub-feature and the second sub-feature is low, the result image obtained based on the second image feature and the third image feature obtained by feature transformation can retain only some image features in the original facial image that are highly similar to the first image feature, that is, the result image can retain only the facial features with higher quality in the original facial image, thereby improving the beautification effect of the result image.

[0134] In step S406, the server determines the editing degree corresponding to the second sub-feature pair based on the second sub-feature pair through the image editing module in the image processing model, each second sub-feature pair includes sub-features at corresponding positions in the second image feature and the third image feature, and the editing degree is used to indicate the degree of editing of the second sub-feature by the third sub-feature in the second sub-feature pair, the third sub-feature is the sub-feature at the corresponding position in the third image feature, and the second sub-feature is the sub-feature at the corresponding position in the second image feature, and the editing degree is proportional to the difference between the second sub-feature and the third sub-feature.

[0135] In the disclosed embodiment, the second image feature includes multiple sub-features, and the third image feature also includes multiple sub-features. Since the multiple sub-features in the second image feature correspond one-to-one to the multiple sub-features in the third image feature, the server can determine multiple second sub-feature pairs based on the second image feature and the third image feature. Each second sub-feature pair includes sub-features at corresponding positions in the second image feature and the third image feature.

[0136] The image editing module also includes a decoder for decoding the image feature and outputting the image represented by the image feature. Accordingly, the server can decode the second image feature and the third image feature obtained by feature transformation through the image editing module in the image processing model to obtain a result image. In the process of decoding the second image feature and the third image feature obtained by feature transformation, the server can first determine the editing degree corresponding to multiple second sub-feature pairs, and then decode the second image feature based on the editing degree to obtain the result image. Accordingly, when determining the editing degree, the server can determine the editing degree corresponding to each second sub-feature pair. That is, for any second sub-feature, the server can determine the editing degree corresponding to the second sub-feature pair based on the two sub-features in the second sub-feature pair through the image editing module in the image processing model. Among them, the editing degree is used to indicate the editing degree of the third sub-feature in the second sub-feature pair to the second sub-feature, the third sub-feature is the sub-feature at the corresponding position in the third image feature, and the second sub-feature is the sub-feature at the corresponding position in the second image feature.

[0137] In step S407, the server decodes the second image feature through the image editing module in the image processing model based on the editing degrees corresponding to the plurality of second sub-feature pairs to obtain a result image.

[0138] In the embodiment of the present disclosure, after determining the editing degrees corresponding to multiple second sub-feature pairs, the server can continue to decode the second image features through the image editing module based on the editing degrees corresponding to multiple second sub-feature pairs, and obtain an image after editing the original facial image based on at least one reference facial image, that is, a result image.

[0139] In addition to including multiple sub-features corresponding to the third image feature one by one, that is, multiple facial features of the object in the original facial image, the second image feature also includes other image features other than the multiple facial features, such as background features of the original facial image, expression features of the object, posture features, etc. Correspondingly, in the process of decoding the second image feature based on the corresponding editing degrees of the multiple second sub-features, the server can first decode the other image features other than the multiple facial features in the second image feature through the image editing module to obtain other image parts other than the object's face in the result image. Then the server can edit the multiple sub-features in the second image feature based on the corresponding editing degrees of the multiple second sub-features to obtain multiple facial features of the object in the result image. Finally, the server can decode the edited multiple facial features through the image editing module to obtain the object's face in the result image. By decoding the second image features based on the corresponding editing degrees of multiple second sub-features, the original facial image can be edited according to the editing degree of the third image features obtained by feature transformation, so that the edited result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.

[0140] In some embodiments, in order to ensure the accuracy of the image processing model, the server needs to train the image processing model. The training process of the image processing model includes the following steps:

[0141] (1) The server processes at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image. The sample reference facial image and the sample original facial image are both images of the face of any object. The object included in the sample reference facial image may be the same as or different from the object included in the sample original facial image, and the embodiments of the present disclosure are not limited to this. The prediction result image is an image obtained by editing the sample original image based on at least one sample reference facial image, that is, the object in the prediction result image is the same as the object in the sample original facial image, and the prediction result image retains some image features of the original facial image while including the image features of at least one sample reference facial image.

[0142] In some embodiments, the process of the server processing at least one sample reference facial image and a sample original facial image through an image processing model includes: processing at least one sample reference facial image through an image processing model to obtain a first sample image feature; extracting features from the sample original image through an image processing model to obtain a second sample image feature; performing feature transformation on the first sample image feature based on the second sample image feature through the image processing model to obtain a third sample image feature, wherein the feature transformation is used to update the editing degree of the first sample image feature to the sample original facial image; decoding the second sample image feature and the third sample image feature through the image processing model to obtain a prediction result image. The first sample image feature is used to represent at least one sample reference facial image. The second sample image feature is used to represent the sample original facial image. In order to prevent the facial features of the object in the prediction result image from being completely identical to those of the at least one sample reference facial image, resulting in the prediction result image lacking the facial features of the sample original facial image. The server can perform feature transformation on the first sample image feature based on the second sample image feature to update the editing degree of the first sample image feature to the sample original facial image. The third sample image feature is the first sample image feature after feature transformation. By performing feature transformation on the first sample image feature based on the second sample image feature, the editing degree of the sample original facial image by the first sample image feature can be updated, and then when the image features of the sample original facial image and the third sample image features obtained by the feature transformation are decoded, the decoded prediction result image can retain some image features of the sample original facial image, that is, the prediction result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the prediction result image output by the image processing model.

[0143] In some embodiments, the process of the server performing feature transformation on the first sample image feature based on the second sample image feature through the image processing model includes: determining the similarity corresponding to the sample sub-feature pairs based on the sample sub-feature pairs in the first sample image feature and the sample second image feature through the image processing model, each sample sub-feature pair includes the sample sub-features at the corresponding positions in the first sample image feature and the second sample image feature, and the similarity is used to indicate the degree of similarity between the two sample sub-features in the sample sub-feature pair; based on the similarities corresponding to the multiple sample sub-feature pairs, performing feature transformation on the first sample image feature to obtain the third sample image feature. The first sample image feature includes multiple sample sub-features, and the second sample image feature also includes multiple sample sub-features, and the corresponding positions of the multiple sample sub-features in the first sample image feature or the second sample image feature are different. Since the multiple sample sub-features in the first sample image feature correspond to the multiple sample sub-features in the second sample image feature one-to-one, the server can determine multiple sample sub-feature pairs based on the first sample image feature and the second sample image feature. Each sample sub-feature pair includes the sample sub-features at the corresponding positions in the first sample image feature and the second sample image feature. In the process of performing feature transformation on the first sample image feature based on the second sample image feature, the server can determine the corresponding similarity of the sample sub-feature pair based on the two sample sub-features in the sample sub-feature pair through the image processing model, and then can perform feature transformation on the first sample image feature based on the corresponding similarity of multiple sample sub-feature pairs to obtain the third sample image feature. By performing feature transformation on the first sample image feature based on the corresponding similarity of multiple sample sub-feature pairs, the feature transformation module can update the editing degree of the sample sub-feature at the corresponding position in the first sample image feature on the sample original facial image based on the similarity between the sub-features at the corresponding positions in the first image feature and the second image feature, so that the predicted result image obtained based on the image feature of the sample original facial image and the third image feature obtained by feature transformation can selectively retain part of the image features in the sample original facial image, that is, the predicted result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the predicted result image.

[0144] (2) Since the multiple facial features in the predicted result image must be consistent with the multiple facial features (third sample image features) of at least one sample reference facial image, the server can determine the first training loss based on the at least one sample reference facial image and the predicted result image. The first training loss is used to indicate the difference between the at least one sample reference facial image and the predicted result image, that is, to indicate the difference between the multiple facial features in the predicted result image and the multiple facial features of the at least one sample reference facial image. The multiple facial features include skin color and texture features, facial features and makeup features. Accordingly, the first training loss may include skin color and texture loss, facial features and makeup loss. The skin color and texture loss is used to indicate the difference in skin color and texture between the sample reference facial image and the predicted result image, the facial features and face shape loss is used to indicate the difference in facial features and face shape between the sample reference facial image and the predicted result image, and the makeup loss is used to indicate the difference in makeup between the sample reference facial image and the predicted result image.

[0145] (3) Since other image features in the predicted result image except facial features must be consistent with the sample original facial image, the server can determine the second training loss based on the sample original facial image and the predicted result image. The second training loss is used to indicate the difference between the sample original facial image and the predicted result image, that is, to indicate the difference between other image features in the predicted result image and other image features in the sample original facial image. Other image features include image background features, expression features of objects in the image, posture features, etc. Accordingly, the second training loss may include background consistency loss, expression loss, posture loss, etc. Background consistency loss is used to indicate the difference in image background between the sample original facial image and the predicted result image, expression loss is used to indicate the difference in object expression between the sample original facial image and the predicted result image, and posture loss is used to indicate the difference in object posture between the sample original facial image and the predicted result image.

[0146] (4) The server trains the image processing model based on the first training loss and the second training loss. Whether the image features of the predicted result image are consistent with the image features of at least one sample reference facial image can reflect the accuracy of the image processing model. Therefore, the first training loss and the second training loss can be used to train the image processing model. By training the image processing model based on the first training loss and the second training loss, the facial features in the result image output by the trained image processing model can be made consistent with the reference facial image after feature transformation, and other image features can be made consistent with the original facial image. As a result, the result image can include not only the image features of the reference facial image, but also retain the image features of the original facial image, thereby improving the accuracy of the image processing model.

[0147] The present disclosure provides a facial image processing method, which can update the editing degree of the image features of the at least one reference facial image to the original facial image by performing feature transformation on the image features of the original facial image based on the image features of the original facial image, and then when decoding the image features of the original facial image and the third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, so that the beautification effect of the processed facial image can be improved while ensuring the processing efficiency of the facial image.

[0148] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0149] Figure 5 is a block diagram of a facial image processing device according to an exemplary embodiment. Figure 5 The device includes: a processing unit 501, a feature extraction unit 502, a feature transformation unit 503 and a decoding unit 504.

[0150] The processing unit 501 is configured to process at least one reference facial image to obtain a first image feature;

[0151] The feature extraction unit 502 is configured to extract features from the original facial image to obtain second image features;

[0152] A feature transformation unit 503 is configured to perform a feature transformation on the first image feature based on the second image feature to obtain a third image feature, wherein the feature transformation is used to update the editing degree of the first image feature on the original facial image;

[0153] The decoding unit 504 is configured to decode the second image feature and the third image feature to obtain a result image.

[0154] In some embodiments, the processing unit 501 includes:

[0155] The extraction subunit 5011 is configured to perform feature extraction on at least one reference facial image respectively to obtain image features of at least one reference facial image;

[0156] The fusion subunit 5012 is configured to perform weighted fusion on the image features of the at least one reference facial image based on the feature fusion weight corresponding to the at least one reference facial image to obtain a first image feature.

[0157] In some embodiments, the processing unit 501 further includes:

[0158] The first determination subunit 5013 is configured to determine, for any reference facial image, a facial angle and facial clarity of the reference facial image based on the image features of the reference facial image, wherein the facial angle is used to indicate the degree of deflection of the face in the reference facial image, and the facial clarity is used to indicate the degree of clarity of the face in the reference facial image; and determine a feature fusion weight corresponding to the reference facial image based on at least one of the facial angle and the facial clarity.

[0159] In some embodiments, the feature conversion unit 503 includes:

[0160] The second determining subunit 5031 is configured to determine the similarity corresponding to the first sub-feature pair based on the first sub-feature pair, each first sub-feature pair includes sub-features at corresponding positions in the first image feature and the second image feature, and the similarity is used to indicate the degree of similarity between the two sub-features in the first sub-feature pair;

[0161] The transformation subunit 5032 is configured to perform feature transformation on the first image feature based on the similarities corresponding to the plurality of first sub-feature pairs to obtain a third image feature.

[0162] In some embodiments, the first sub-feature pair includes a first sub-feature from the first image feature and a second sub-feature from the second image feature;

[0163] The transformation subunit 5032 is configured to update the feature transformation weight of the first sub-feature for any first sub-feature pair based on the corresponding similarity of the first sub-feature pair, where the feature transformation weight is used to indicate the degree of editing of the original facial image by the first sub-feature; based on the feature transformation weight, weight the first sub-feature to obtain a third sub-feature, where the third sub-feature is a sub-feature at a corresponding position in the third image feature.

[0164] In some embodiments, the transformation subunit 5032 is configured to reduce the feature transformation weight when the similarity corresponding to the first sub-feature pair is greater than a first threshold; and to increase the feature transformation weight when the similarity corresponding to the first sub-feature pair is less than a second threshold, and the second threshold is not greater than the first threshold.

[0165] In some embodiments, the decoding unit 504 is configured to determine the editing degree corresponding to the second sub-feature pair based on the second sub-feature pair, each second sub-feature pair includes sub-features at corresponding positions in the second image feature and the third image feature, and the editing degree is used to indicate the editing degree of the third sub-feature in the second sub-feature pair to the second sub-feature, the third sub-feature is the sub-feature at the corresponding position in the third image feature, and the second sub-feature is the sub-feature at the corresponding position in the second image feature, and the editing degree is proportional to the difference between the second sub-feature and the third sub-feature; based on the editing degrees corresponding to multiple second sub-feature pairs, the second image feature is decoded to obtain a result image.

[0166] The present disclosure provides a facial image processing device, which can update the editing degree of the image features of at least one reference facial image to the original facial image by performing feature transformation on the image features of at least one reference facial image based on the image features of the original facial image, and then when decoding the image features of the original facial image and the third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image while improving the beautification effect of the processed facial image.

[0167] It should be noted that the facial image processing device provided in the above embodiment only uses the division of the above functional units as an example to illustrate when processing facial images. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the facial image processing device provided in the above embodiment and the facial image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0168] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0169] Figure 6 is a block diagram of a model training device according to an exemplary embodiment. Figure 6 The device includes: a processing unit 601, a first determination unit 602, a second determination unit 603 and a training unit 604.

[0170] The processing unit 601 is configured to process at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image;

[0171] A first determining unit 602 is configured to determine a first training loss based on at least one sample reference facial image and the prediction result image, wherein the first training loss is used to indicate a difference between the at least one sample reference facial image and the prediction result image;

[0172] A second determining unit 603 is configured to determine a second training loss based on the sample original facial image and the prediction result image, where the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;

[0173] The training unit 604 is configured to train the image processing model based on the first training loss and the second training loss.

[0174] In some embodiments, the processing unit 601 is configured to process at least one sample reference facial image through an image processing model to obtain a first sample image feature; perform feature extraction on the sample original image through the image processing model to obtain a second sample image feature; perform feature transformation on the first sample image feature based on the second sample image feature through the image processing model to obtain a third sample image feature, and the feature transformation is used to update the degree of editing of the sample original facial image by the first sample image feature; and decode the second sample image feature and the third sample image feature through the image processing model to obtain a predicted result image.

[0175] In some embodiments, the processing unit 601 is configured to determine, through an image processing model, the similarities corresponding to the sample sub-feature pairs based on the sample sub-feature pairs in the first sample image feature and the sample second image feature, each sample sub-feature pair includes the sample sub-features at corresponding positions in the first sample image feature and the second sample image feature, and the similarity is used to indicate the degree of similarity between the two sample sub-features in the sample sub-feature pair; based on the similarities corresponding to multiple sample sub-feature pairs, perform feature transformation on the first sample image feature to obtain a third sample image feature.

[0176] The present disclosure implements and provides a model training device. By training the image processing model based on a first training loss and a second training loss, the facial features in the result image output by the trained image processing model can be made consistent with the reference facial image after feature transformation, and other image features can be made consistent with the original facial image. As a result, the result image can include not only the image features of the reference facial image, but also retain the image features of the original facial image, thereby improving the accuracy of the image processing model.

[0177] It should be noted that the model training device provided in the above embodiment only uses the division of the above functional units as an example when training the image processing model. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the model training device provided in the above embodiment belongs to the same concept as the model training method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0178] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0179] When an electronic device is provided as a terminal, Figure 7 FIG. 1 is a block diagram of a terminal according to an exemplary embodiment. Figure 7 As shown, the terminal 700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a notebook computer or a desktop computer. The terminal 700 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0180] Typically, the terminal 700 includes a processor 701 and a memory 702 .

[0181] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0182] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the facial image processing method or model training method provided in the method embodiment of the present disclosure.

[0183] In some embodiments, the terminal 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709.

[0184] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0185] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0186] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the terminal 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the terminal 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0187] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0188] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0189] The power supply 708 is used to power various components in the terminal 700. The power supply 708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0190] In some embodiments, the terminal 700 further includes one or more sensors 709 , including but not limited to: an acceleration sensor 7010 , a gyroscope sensor 7011 , a pressure sensor 7012 , an optical sensor 7013 , and a proximity sensor 7014 .

[0191] The acceleration sensor 7010 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 700. For example, the acceleration sensor 7010 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 701 can control the display screen 705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 7010. The acceleration sensor 7010 can also be used for collecting game or user motion data.

[0192] The gyro sensor 7011 can detect the body direction and rotation angle of the terminal 700, and the gyro sensor 7011 can cooperate with the acceleration sensor 7010 to collect the user's 3D actions on the terminal 700. The processor 701 can implement the following functions based on the data collected by the gyro sensor 7011: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0193] The pressure sensor 7012 can be set on the side frame of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 7012 is set on the side frame of the terminal 700, it can detect the user's holding signal of the terminal 700, and the processor 701 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 7012. When the pressure sensor 7012 is set on the lower layer of the display screen 705, the processor 701 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0194] The optical sensor 7013 is used to collect the ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 according to the ambient light intensity collected by the optical sensor 7013. Optionally, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is reduced. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 7013.

[0195] The proximity sensor 7014, also called a distance sensor, is disposed on the front panel of the terminal 700. The proximity sensor 7014 is used to collect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 7014 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from the screen-on state to the screen-off state; when the proximity sensor 7014 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from the screen-off state to the screen-on state.

[0196] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 700, and the terminal 700 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0197] When the electronic device is provided as a server, Figure 8It is a block diagram of a server according to an exemplary embodiment. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one program code, and the at least one program code is loaded and executed by the processor 801 to implement the facial image processing method or model training method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 800 may also include other components for implementing device functions, which will not be described in detail here.

[0198] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 or a memory 802 including instructions, and the instructions can be executed by the processor 701 of the terminal 700 or the processor 801 of the server 800 to complete the above-mentioned facial image processing method, or the above-mentioned model training method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0199] A computer program product includes a computer program, which implements the above-mentioned facial image processing method or the above-mentioned model training method when executed by a processor.

[0200] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0201] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A facial image processing method, It is characterized in that The method comprises: Processing at least one reference facial image to obtain a first image feature; Extracting features from the original facial image to obtain second image features; Based on the second image feature, performing a feature transformation on the first image feature to obtain a third image feature, wherein the feature transformation is used to update the editing degree of the first image feature on the original facial image; The second image feature and the third image feature are decoded to obtain a result image.

2. The facial image processing method according to claim 1, It is characterized in that The step of processing at least one reference facial image to obtain a first image feature includes: Extracting features from the at least one reference facial image respectively to obtain image features of the at least one reference facial image; Based on the feature fusion weight corresponding to the at least one reference facial image, the image features of the at least one reference facial image are weightedly fused to obtain the first image feature.

3. The facial image processing method according to claim 2, It is characterized in that The method further comprises: For any reference facial image, determining a facial angle and facial clarity of the reference facial image based on image features of the reference facial image, wherein the facial angle is used to indicate a degree of deflection of a face in the reference facial image, and the facial clarity is used to indicate a degree of clarity of a face in the reference facial image; Based on at least one of the facial angle and the facial definition, a feature fusion weight corresponding to the reference facial image is determined.

4. The facial image processing method according to claim 1, It is characterized in that The step of performing feature transformation on the first image feature based on the second image feature to obtain a third image feature includes: Based on the first sub-feature pairs, determining the similarity corresponding to the first sub-feature pairs, each first sub-feature pair includes sub-features at corresponding positions in the first image feature and the second image feature, and the similarity is used to indicate the degree of similarity between two sub-features in the first sub-feature pair; Based on the similarities corresponding to the plurality of first sub-feature pairs, feature transformation is performed on the first image feature to obtain the third image feature.

5. The facial image processing method according to claim 4, It is characterized in that The first sub-feature pair includes a first sub-feature from the first image feature and a second sub-feature from the second image feature; The step of performing feature transformation on the first image feature based on the similarities corresponding to the plurality of first sub-feature pairs to obtain the third image feature includes: For any first sub-feature pair, based on the similarity corresponding to the first sub-feature pair, updating the feature transformation weight of the first sub-feature, the feature transformation weight being used to indicate the degree of editing of the original facial image by the first sub-feature; Based on the feature transformation weight, the first sub-feature is weighted to obtain a third sub-feature, where the third sub-feature is a sub-feature at a corresponding position in the third image feature.

6. The facial image processing method according to claim 5, It is characterized in that The updating of the feature transformation weight of the first sub-feature based on the similarity corresponding to the first sub-feature pair includes: When the similarity corresponding to the first sub-feature pair is greater than a first threshold, reducing the feature transformation weight; When the similarity corresponding to the first sub-feature pair is less than a second threshold, the feature transformation weight is increased, and the second threshold is not greater than the first threshold.

7. The facial image processing method according to claim 1, It is characterized in that The decoding of the second image feature and the third image feature to obtain a result image includes: Based on the second sub-feature pairs, determining the editing degree corresponding to the second sub-feature pairs, each second sub-feature pair includes the sub-features at corresponding positions in the second image feature and the third image feature, the editing degree is used to indicate the editing degree of the second sub-feature by the third sub-feature in the second sub-feature pair, the third sub-feature is the sub-feature at the corresponding position in the third image feature, the second sub-feature is the sub-feature at the corresponding position in the second image feature, and the editing degree is proportional to the difference between the second sub-feature and the third sub-feature; Based on the editing degrees corresponding to the plurality of second sub-feature pairs, the second image feature is decoded to obtain the result image.

8. A model training method, It is characterized in that The method comprises: Processing at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image; Determining a first training loss based on the at least one sample reference facial image and the prediction result image, the first training loss being used to indicate a difference between the at least one sample reference facial image and the prediction result image; Determining a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image; The image processing model is trained based on the first training loss and the second training loss.

9. The model training method according to claim 8, It is characterized in that The method of processing at least one sample reference facial image and a sample original facial image by an image processing model to obtain a prediction result image includes: Processing the at least one sample reference facial image by the image processing model to obtain a first sample image feature; By using the image processing model, feature extraction is performed on the sample original image to obtain second sample image features; By means of the image processing model, based on the second sample image feature, a feature transformation is performed on the first sample image feature to obtain a third sample image feature, wherein the feature transformation is used to update the editing degree of the first sample image feature on the sample original facial image; The second sample image feature and the third sample image feature are decoded by the image processing model to obtain the prediction result image.

10. The model training method according to claim 9, It is characterized in that The step of performing feature transformation on the first sample image feature based on the second sample image feature through the image processing model to obtain the third sample image feature includes: Determining, by the image processing model, based on the sample sub-feature pairs in the first sample image feature and the sample second image feature, the similarity corresponding to the sample sub-feature pairs, each sample sub-feature pair comprising sample sub-features at corresponding positions in the first sample image feature and the second sample image feature, and the similarity is used to indicate the degree of similarity between the two sample sub-features in the sample sub-feature pair; Based on the similarities corresponding to the plurality of sample sub-feature pairs, feature transformation is performed on the first sample image feature to obtain the third sample image feature.

11. A facial image processing device, It is characterized in that The device comprises: a processing unit configured to process at least one reference facial image to obtain a first image feature; A feature extraction unit is configured to extract features from the original facial image to obtain second image features; a feature transformation unit configured to perform feature transformation on the first image feature based on the second image feature to obtain a third image feature, wherein the feature transformation is used to update the editing degree of the first image feature on the original facial image; A decoding unit is configured to decode the second image feature and the third image feature to obtain a result image.

12. A model training device, It is characterized in that The device comprises: A processing unit is configured to process at least one sample reference facial image and a sample original facial image through an image processing model to obtain a prediction result image; A first determining unit is configured to determine a first training loss based on the at least one sample reference facial image and the prediction result image, wherein the first training loss is used to indicate a difference between the at least one sample reference facial image and the prediction result image; A second determining unit is configured to determine a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image; A training unit is configured to train the image processing model based on the first training loss and the second training loss.

13. An electronic device, It is characterized in that The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the facial image processing method as described in any one of claims 1 to 7, or the model training method as described in any one of claims 8 to 10.

14. A computer-readable storage medium, It is characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the facial image processing method as described in any one of claims 1 to 7, or the model training method as described in any one of claims 8 to 10.

15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, it implements the facial image processing method described in any one of claims 1 to 7, or the model training method described in any one of claims 8 to 10.