Facial image processing method and device, model training method and device and electronic equipment
By extracting and transforming multiple reference facial images, the editing degree of the original facial image is updated, and the problem of poor beautification effect in the existing technology is solved, efficient beautification and facial features are preserved.
Patent Information
- Application Number
- CN202411009612.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
When existing facial image processing technology beautifies facial images, it is difficult to maintain the original facial features, resulting in low processing efficiency and poor beautification effect.
By extracting and transforming multiple reference facial image pairs, the editing degree of the original facial image is updated, ensuring that the result image retains the original facial features while improving the beautification effect.
While ensuring the efficiency of facial image processing, the beautification effect of the processed facial image is improved, so that the resulting image retains the characteristics of the original facial.
Smart Images

Figure CN118941440A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a facial image processing method, a model training method, a device and an electronic device. Background Art
[0002] With the development of Internet technology and artificial intelligence technology, users can use various virtual tools in applications to process facial images to beautify them. Facial processing includes a variety of processing methods, such as whitening, skin resurfacing, makeup, and face slimming. However, during the facial processing process, users are required to perform multiple manual operations on facial images, which is a large workload and leads to low image processing efficiency. Therefore, how to improve the image processing efficiency when performing facial processing on facial images is a technical problem that needs to be solved.
[0003] In the related art, taking a facial processing method as an example, a makeup transfer method can be used to process a facial image. This method can transfer the makeup of a person in a reference facial image to a facial image provided by a user through a makeup transfer model, so that a processed facial image can be quickly obtained.
[0004] However, when performing facial processing, the above solution will replace all the makeup of the facial image provided by the user with the makeup of the person in the reference image, resulting in a poor beautification effect of the processed facial image. Summary of the invention
[0005] The present disclosure provides a facial image processing method, a model training method, a device and an electronic device, which can make the result image have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image and improving the beautification effect of the processed facial image. The technical solution of the present disclosure is as follows:
[0006] According to one aspect of an embodiment of the present disclosure, a facial image processing method is provided, including:
[0007] Processing a plurality of reference facial image pairs to obtain a plurality of first image features, each reference facial image pair comprising two reference facial images from the same subject, the first image features being used to represent a difference between the two reference facial images in the reference facial image pair in a corresponding facial feature dimension, and different first image features corresponding to different facial feature dimensions;
[0008] Extracting features from the original facial image to obtain second image features;
[0009] Based on the second image feature, performing feature transformation on the plurality of first image features to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image;
[0010] The second image feature and the plurality of third image features are decoded to obtain a result image.
[0011] According to one aspect of an embodiment of the present disclosure, a model training method is provided, including:
[0012] Processing a plurality of sample reference facial image pairs and sample original facial images through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object;
[0013] For any sample reference facial image pair, determining a first training loss based on the sample reference facial image pair and the prediction result image, wherein the first training loss is used to indicate a difference between the sample reference facial image pair and the prediction result image;
[0014] Determining a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;
[0015] The image processing model is trained based on multiple first training losses and the second training losses.
[0016] According to another aspect of an embodiment of the present disclosure, there is provided a facial image processing apparatus, comprising:
[0017] a processing unit configured to process a plurality of reference facial image pairs to obtain a plurality of first image features, each reference facial image pair comprising two reference facial images from the same subject, the first image features being used to represent a difference between the two reference facial images in the reference facial image pair in a corresponding facial feature dimension, and different first image features corresponding to different facial feature dimensions;
[0018] A feature extraction unit is configured to extract features from the original facial image to obtain second image features;
[0019] a feature transformation unit configured to perform feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image;
[0020] A decoding unit is configured to decode the second image feature and the plurality of third image features to obtain a result image.
[0021] In some embodiments, the reference facial image pair includes a first reference facial image and a second reference facial image, the first reference facial image is a reference facial image in which facial modification has not been performed on the subject, and the second reference facial image is a reference facial image in which facial modification has been performed on the subject;
[0022] The processing unit comprises:
[0023] an extraction subunit, configured to, for any reference facial image pair, perform feature extraction on the first reference facial image and the second reference facial image, respectively, to obtain a fourth image feature and a fifth image feature, wherein the fourth image feature is an image feature of the first reference facial image in a corresponding facial feature dimension, and the fifth image feature is an image feature of the second reference facial image in a corresponding facial feature dimension;
[0024] The processing subunit is configured to subtract the fourth image feature from the fifth image feature to obtain a first image feature corresponding to the reference facial image pair.
[0025] In some embodiments, the feature transformation unit is also configured to receive instructions for setting weights for multiple facial feature dimensions; for any facial feature dimension, in response to instructions for setting weights for the facial feature dimension, based on the weight and the second image feature, a feature transformation is performed on the first image feature corresponding to the facial feature dimension to obtain a third image feature.
[0026] In some embodiments, the feature conversion unit includes:
[0027] a determination subunit configured to determine, for any first image feature, a similarity corresponding to the first image feature based on the first image feature and the second image feature, wherein the similarity is used to indicate a degree of similarity between the first image feature and the second image feature;
[0028] The transformation subunit is configured to perform feature transformation on the first image feature based on the similarity to obtain the third image feature.
[0029] In some embodiments, the transformation subunit is configured to update a feature transformation weight of the first image feature based on the similarity, wherein the feature transformation weight is used to indicate the degree of editing of the original facial image by the first image feature; and weight the first image feature based on the feature transformation weight to obtain the third image feature.
[0030] In some embodiments, the transformation subunit is configured to reduce the feature transformation weight when the similarity is greater than a first threshold; and increase the feature transformation weight when the similarity is less than a second threshold, and the second threshold is not greater than the first threshold.
[0031] In some embodiments, the decoding unit is configured to determine, for any third image feature, a degree of editing corresponding to the third image feature based on the third image feature and the second image feature, the degree of editing being used to indicate the degree of editing of the second image feature by the third image feature, and the degree of editing being proportional to the difference between the third image feature and the second image feature; and decode the second image feature based on the degrees of editing corresponding to the multiple third image features to obtain the result image.
[0032] According to another aspect of an embodiment of the present disclosure, a model training device is provided, including:
[0033] A processing unit is configured to process a plurality of sample reference facial image pairs and a sample original facial image through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object;
[0034] A first determining unit is configured to determine, for any sample reference facial image pair, a first training loss based on the sample reference facial image pair and the prediction result image, wherein the first training loss is used to indicate a difference between the sample reference facial image pair and the prediction result image;
[0035] A second determining unit is configured to determine a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;
[0036] A training unit is configured to train the image processing model based on a plurality of the first training losses and the second training losses.
[0037] In some embodiments, the processing unit is configured to process the multiple sample reference facial image pairs through the image processing model to obtain multiple first sample image features, where the first sample image features are used to represent the difference between the two sample reference facial images in the sample reference facial image pair in the corresponding facial feature dimensions, and different first sample image features correspond to different facial feature dimensions; to extract features from the sample original image through the image processing model to obtain second sample image features; to perform feature transformation on the multiple first sample image features based on the second sample image features through the image processing model to obtain multiple third sample image features, where the feature transformation is used to update the editing degree of the sample original facial image by the first sample image features; and to decode the second sample image features and the multiple third sample image features through the image processing model to obtain the predicted result image.
[0038] In some embodiments, the processing unit is configured to determine, for any first sample image feature, through the image processing model, the similarity corresponding to the first sample image feature based on the first sample image feature and the sample second image feature, wherein the similarity is used to indicate the degree of similarity between the first sample image feature and the second sample image feature; based on the similarity, perform feature transformation on the first sample image feature to obtain the third sample image feature.
[0039] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, the electronic device comprising:
[0040] one or more processors;
[0041] A memory for storing program codes executable by the processor;
[0042] Among them, the processor is configured to execute the program code to implement the above-mentioned facial image processing method or the above-mentioned model training method.
[0043] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device can perform the above-mentioned facial image processing method or the above-mentioned model training method.
[0044] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned facial image processing method or the above-mentioned model training method when executed by a processor.
[0045] The disclosed embodiment provides a facial image processing method, which can update the editing degree of the original facial image by the multiple first image features corresponding to the multiple reference facial image pairs by performing feature transformation on the multiple first image features corresponding to the multiple reference facial image pairs based on the image features of the original facial image, and then when decoding the image features of the original facial image and the multiple third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image while improving the beautification effect of the processed facial image.
[0046] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0048] Figure 1 The figure is a schematic diagram of an implementation environment of a facial image processing method according to an exemplary embodiment.
[0049] Figure 2 The figure is a flowchart of a facial image processing method according to an exemplary embodiment.
[0050] Figure 3 It is a structural schematic diagram of an image processing model according to an exemplary embodiment.
[0051] Figure 4 The figure is a flowchart of another facial image processing method according to an exemplary embodiment.
[0052] Figure 5 The invention is a block diagram of a facial image processing apparatus according to an exemplary embodiment.
[0053] Figure 6 It is a block diagram of a model training device according to an exemplary embodiment.
[0054] Figure 7 It is a block diagram of a terminal according to an exemplary embodiment.
[0055] Figure 8 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION
[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0058] The user information involved in this disclosure may be information authorized by the user or fully authorized by all parties. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the facial images involved in this application are all obtained with full authorization.
[0059] The facial image processing method provided by the embodiments of the present disclosure can be performed by an electronic device. In some embodiments, the electronic device can be provided as a terminal or a server. When the electronic device is provided as a terminal, the terminal can implement the operations performed by the facial image processing method; when it is provided as a server, the server can implement the operations performed by the facial image processing method. The server can receive a facial image authorized by the terminal, and the server processes the facial image based on the image processing model; the server and the terminal can also interact to implement the operations performed by the facial image processing method; the terminal can also send an image processing request to the server, and the server performs image processing, and then feeds back the result of the image processing to the terminal, and the terminal outputs the result of the image processing.
[0060] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a facial image processing method according to an exemplary embodiment. Figure 1 , the implementation environment specifically includes: a terminal 101 and a server 102.
[0061] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and a user may log in to the application through the terminal 101 to obtain the services provided by the application. The server 102 may be a background server of the application, and is used to provide background services for the application. The terminal 101 may be connected to the server 102 via a wireless network or a wired network, and the facial image to be processed may be sent to the server 102.
[0062] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be only a few, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.
[0063] The server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. The server 102 can receive the facial image sent by the terminal 101, and return the result image obtained by processing the facial image to the terminal 101 for the user to view. Optionally, the number of the above servers can be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.
[0064] Figure 2 is a flowchart of a facial image processing method according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:
[0065] In step S201, the electronic device processes multiple reference facial image pairs to obtain multiple first image features, each reference facial image pair includes two reference facial images from the same object, and the first image features are used to represent the difference between the two reference facial images in the reference facial image pair in corresponding facial feature dimensions, and different first image features correspond to different facial feature dimensions.
[0066] In the embodiments of the present disclosure, the multiple reference facial image pairs may be the same reference facial image pairs or different reference facial image pairs, and the embodiments of the present disclosure are not limited in this regard. For any reference facial image pair, the electronic device can process the reference facial image pair to obtain a first image feature corresponding to the reference facial image pair. The first image feature is used to represent the difference between the two reference facial images in the reference facial image pair in the corresponding facial feature dimension. The facial feature dimension may be any one of dimensions such as skin color, makeup, facial features, and face shape. The object in the reference facial image may be a person or an animal, and the embodiments of the present disclosure are not limited in this regard.
[0067] In step S202, the electronic device extracts features from the original facial image to obtain second image features.
[0068] In the disclosed embodiment, the original facial image may be a facial image of the user himself, or a facial image taken by the user for another object, or a facial image of any object downloaded by the user from the network, and the disclosed embodiment does not limit this. The object included in the original facial image may be the same as or different from the object included in the reference facial image in the reference facial image pair, and the disclosed embodiment does not limit this. The electronic device can extract features from the original facial image to obtain second image features for representing the original facial image.
[0069] In step S203, the electronic device performs feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image.
[0070] In the disclosed embodiment, since the electronic device needs to edit the original facial image based on the reference facial image in the plurality of reference facial image pairs to obtain the beautified original facial image, the electronic device can edit the second image feature based on the plurality of first image features to obtain the image features of the beautified original facial image. However, if the image features of the original facial image are all replaced with the image features of the reference facial image during the editing process, the beautified original facial image will lack the facial features of the original facial image. In order to solve this problem, the electronic device can perform feature transformation on the plurality of first image features based on the second image feature to update the editing degree of the plurality of first image features on the original facial image. The third image feature is the first image feature after the feature transformation.
[0071] In step S204, the electronic device decodes the second image feature and the plurality of third image features to obtain a result image.
[0072] In the disclosed embodiment, the result image is an image obtained by editing the original facial image based on the reference facial image in the plurality of reference facial image pairs. After performing feature transformation on the plurality of first image features to obtain the plurality of third image features, the electronic device can decode the second image features and the plurality of third image features to obtain the result image. The result image includes the image features of the reference facial image while retaining some image features of the original facial image.
[0073] The present disclosure provides a facial image processing method, which can update the editing degree of the original facial image by the multiple first image features corresponding to the multiple reference facial image pairs by performing feature transformation on the multiple first image features corresponding to the multiple reference facial image pairs based on the image features of the original facial image, and then when decoding the image features of the original facial image and the multiple third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, so that the beautification effect of the processed facial image can be improved while ensuring the processing efficiency of the facial image.
[0074] In some embodiments, the reference facial image pair includes a first reference facial image and a second reference facial image, the first reference facial image being a reference facial image in which facial modification has not been performed on the subject, and the second reference facial image being a reference facial image in which facial modification has been performed on the subject;
[0075] Processing a plurality of reference facial image pairs to obtain a plurality of first image features, including:
[0076] For any reference facial image pair, feature extraction is performed on the first reference facial image and the second reference facial image respectively to obtain a fourth image feature and a fifth image feature, wherein the fourth image feature is an image feature of the first reference facial image in a corresponding facial feature dimension, and the fifth image feature is an image feature of the second reference facial image in a corresponding facial feature dimension;
[0077] The fourth image feature is subtracted from the fifth image feature to obtain a first image feature corresponding to the reference facial image pair.
[0078] In the embodiment of the present disclosure, by subtracting the image features of two reference facial images in the reference facial image pair in the corresponding facial feature dimension, a first image feature corresponding to the reference facial image pair is obtained, so that the server can quickly analyze the difference between the two reference facial images in the reference facial image pair in the corresponding facial feature dimension based on the first image feature, thereby quickly determining the facial modification performed on the object in the reference facial image in the corresponding facial feature dimension.
[0079] In some embodiments, the method further comprises:
[0080] receiving an instruction for setting weights of a plurality of facial feature dimensions;
[0081] Based on the second image feature, a plurality of first image features are subjected to feature transformation to obtain a plurality of third image features, including:
[0082] For any facial feature dimension, in response to an instruction to set a weight for the facial feature dimension, a feature transformation is performed on a first image feature corresponding to the facial feature dimension based on the weight and the second image feature to obtain a third image feature.
[0083] In the disclosed embodiment, the first image feature corresponding to each facial feature dimension is weighted based on the weights of multiple facial feature dimensions manually set by the user, so that the user can easily set by manual adjustment which facial feature dimension of the reference facial image in the reference facial image pair is used to influence the final generated result image, as well as the degree of influence of the image features of the reference facial image in the reference facial image pair in the corresponding facial feature dimension on the final generated result image, thereby realizing user-controllable editing of facial images.
[0084] In some embodiments, based on the second image feature, a plurality of first image features are subjected to feature transformation to obtain a plurality of third image features, including:
[0085] For any first image feature, based on the first image feature and the second image feature, determine a similarity corresponding to the first image feature, where the similarity is used to indicate a degree of similarity between the first image feature and the second image feature;
[0086] Based on the similarity, feature transformation is performed on the first image feature to obtain a third image feature.
[0087] In the disclosed embodiment, by performing feature transformation on the first image feature based on the similarity corresponding to the first image feature, the feature processing module can update the degree of editing of the original facial image by the first image feature based on the similarity between the facial features belonging to a specific facial feature dimension in the second image feature and the first image feature, thereby making it possible for a result image obtained based on the image features of the original facial image and multiple third image features obtained by the feature transformation to selectively retain some image features in the original facial image, that is, to make the result image have the facial features of the original facial image, thereby improving the beautification effect of the result image.
[0088] In some embodiments, based on the similarity, performing feature transformation on the first image feature to obtain the third image feature includes:
[0089] Based on the similarity, updating a feature transformation weight of the first image feature, the feature transformation weight being used to indicate a degree of editing of the original facial image by the first image feature;
[0090] The first image feature is weighted based on the feature transformation weight to obtain a third image feature.
[0091] In the disclosed embodiment, by updating the feature transformation weight of the first image feature based on the similarity corresponding to the first image feature, the degree of editing of the original facial image by the first image feature can be changed as the similarity between the first image feature and the second image feature changes, so that the result image obtained based on the image features of the original facial image and multiple third image features obtained by feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.
[0092] In some embodiments, updating the feature transformation weight of the first image feature based on the similarity includes:
[0093] When the similarity is greater than a first threshold, reducing the feature transformation weight;
[0094] When the similarity is less than the second threshold, the feature transformation weight is increased, and the second threshold is not greater than the first threshold.
[0095] In the disclosed embodiment, by reducing the feature transformation weight of the first image feature when the similarity is high, and increasing the feature change weight of the first image feature when the similarity is low, the result image obtained based on the second image feature and the third image feature obtained by feature transformation can retain some image features of the original facial image that have a high similarity with the first image feature, that is, the result image can retain the higher quality facial features in the original facial image, thereby improving the beautification effect of the result image.
[0096] In some embodiments, decoding the second image feature and the plurality of third image features to obtain a result image includes:
[0097] For any third image feature, based on the third image feature and the second image feature, determining an editing degree corresponding to the third image feature, the editing degree being used to indicate the editing degree of the third image feature to the second image feature, and the editing degree being proportional to the difference between the third image feature and the second image feature;
[0098] Based on the editing degrees corresponding to the plurality of third image features, the second image feature is decoded to obtain a result image.
[0099] In the disclosed embodiment, by decoding the second image feature based on the editing degree corresponding to multiple third image features obtained by special transformation, the server can edit the original facial image according to the editing degree of the original facial image by the multiple third image features, so that the edited result image can retain some image features of the original facial image, that is, the result image can have the facial characteristics of the original facial image, thereby improving the beautification effect of the result image.
[0100] In some embodiments, the training process of the image processing model includes:
[0101] Processing a plurality of sample reference facial image pairs and sample original facial images through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object;
[0102] For any sample reference facial image pair, determining a first training loss based on the sample reference facial image pair and the prediction result image, the first training loss being used to indicate a difference between the sample reference facial image pair and the prediction result image;
[0103] Based on the sample original facial image and the predicted result image, determining a second training loss, the second training loss being used to indicate a difference between the sample original facial image and the predicted result image;
[0104] The image processing model is trained based on the plurality of first training losses and the second training losses.
[0105] In the disclosed embodiment, by training the image processing model based on multiple first training losses and second training losses, the image features in the result image output by the trained image processing model can be made consistent not only with the reference facial image, but also with the original facial image, so that the result image can include the image features of the reference facial image while retaining the image features of the original facial image, thereby improving the accuracy of the image processing model.
[0106] In some embodiments, a plurality of sample reference facial image pairs and sample original facial images are processed by an image processing model to obtain a prediction result image, including:
[0107] Processing a plurality of sample reference facial image pairs through an image processing model to obtain a plurality of first sample image features, wherein the first sample image features are used to represent the difference between two sample reference facial images in the sample reference facial image pairs in corresponding facial feature dimensions, and different first sample image features correspond to different facial feature dimensions;
[0108] Through the image processing model, feature extraction is performed on the sample original image to obtain the second sample image feature;
[0109] Through the image processing model, based on the second sample image feature, a plurality of first sample image features are subjected to feature transformation to obtain a plurality of third sample image features, wherein the feature transformation is used to update the editing degree of the first sample image feature to the sample original facial image;
[0110] The second sample image feature and multiple third sample image features are decoded through the image processing model to obtain a prediction result image.
[0111] In the disclosed embodiment, by performing feature transformation on multiple first sample image features based on second sample image features, the editing degree of the sample original facial image by the multiple first sample image features can be updated, and then when the image features of the sample original facial image and the multiple third sample image features obtained by the feature transformation are decoded, the decoded prediction result image can retain some image features of the sample original facial image, that is, the prediction result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the prediction result image output by the image processing model.
[0112] In some embodiments, the image processing model is used to perform feature transformation on the plurality of first sample image features based on the second sample image features to obtain the plurality of third sample image features, including:
[0113] For any first sample image feature, determine the similarity corresponding to the first sample image feature based on the first sample image feature and the sample second image feature through the image processing model, where the similarity is used to indicate the degree of similarity between the first sample image feature and the second sample image feature;
[0114] Based on the similarity, feature transformation is performed on the first sample image feature to obtain a third sample image feature.
[0115] In the disclosed embodiment, by performing feature transformation on the first sample image feature based on the similarity corresponding to the first sample image feature, the feature processing module can update the editing degree of the first sample image feature on the sample original facial image based on the similarity between the facial features belonging to a specific facial feature dimension in the second sample image feature and the first sample image feature, so that the predicted result image obtained based on the image features of the sample original facial image and multiple third sample image features obtained by feature transformation can selectively retain some sample image features in the sample original facial image, that is, the predicted result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the predicted result image.
[0116] Based on the above embodiments, the image processing method can be implemented based on an image processing model. For example, Figure 3 FIG. 1 is a schematic diagram showing a structure of an image processing model according to an exemplary embodiment. Figure 3 As shown in FIG. 1 , the image processing model includes a feature processing module and an image editing module. The feature processing module is used to extract features from the input image and perform feature transformation on the extracted image features. The encoder in the image editing module is used to extract features from the input image. The decoder in the image editing module is used to decode the image features to reconstruct the image. Specifically, Figure 3 As shown, when three reference facial image pairs and an original facial image are input into the image processing model, the three feature extractors in the feature processing module can process the three reference facial image pairs respectively to obtain three first image features. The encoder in the image editing module can extract features from the original facial image to obtain second image features. After obtaining the first image features and the second image features, the feature processing module can also perform feature transformation on the three first image features based on the second image features to obtain three third image features. Finally, the decoder in the image editing module can decode the second image features and the three image features to obtain the result image.
[0117] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 4 FIG. 1 is a flowchart of another facial image processing method according to an exemplary embodiment. Figure 4 , the method comprising:
[0118] In step S401, for any reference facial image pair, the server performs feature extraction on the first reference facial image and the second reference facial image in the reference facial image pair through the feature processing module in the image processing model to obtain a fourth image feature and a fifth image feature. The fourth image feature is the image feature of the first reference facial image in the corresponding facial feature dimension, and the fifth image feature is the image feature of the second reference facial image in the corresponding facial feature dimension.
[0119] In the disclosed embodiment, the feature processing module of the image processing model includes multiple feature extractors, and different feature extractors are used to extract image features of different facial feature dimensions. Accordingly, after obtaining multiple reference facial image pairs uploaded by the user, the server can input the multiple reference facial image pairs into the corresponding feature extractors according to the facial feature dimensions corresponding to the multiple reference facial image pairs set by the user, and the feature extractors process the reference facial image pairs to obtain the first image features corresponding to the reference facial image pairs.
[0120] Since each reference facial image pair includes two reference facial images from the same object, namely, the first reference facial image and the second reference facial image, when processing any reference facial image pair, the server can extract features from the first reference facial image and the second reference facial image respectively through the feature extractor corresponding to the reference facial image pair in the feature processing module to obtain the fourth image feature and the fifth image feature. Since the feature extractor corresponding to the reference facial image pair is used to extract features on the facial feature dimension corresponding to the reference facial image pair, the fourth image feature is the image feature of the first reference facial image on the corresponding facial feature dimension, and the fifth image feature is the image feature of the second reference facial image on the corresponding facial feature dimension. Among them, the first reference facial image is a reference facial image without facial modification of the object, and the second reference facial image is a reference facial image with facial modification of the object. Taking the object as a user as an example, the first reference facial image can be a bare-faced image of the user, and the second reference facial image can be a retouched image of the user.
[0121] In step S402, the server subtracts the fourth image feature from the fifth image feature through the feature processing module in the image processing model to obtain a first image feature corresponding to the reference facial image pair. The first image feature is used to represent the difference between the first reference facial image and the second reference facial image in the corresponding facial feature dimension. Different first image features correspond to different facial feature dimensions.
[0122] In the disclosed embodiment, after the image features of the first reference facial image and the second reference facial image in the corresponding facial feature dimension are extracted by the feature processing module, the server can continue to use the feature processing module to subtract the image features of the first reference facial image in the corresponding facial feature dimension from the image features of the second reference facial image in the corresponding facial feature dimension, that is, to subtract the fourth image feature from the fifth image feature, to obtain the first image feature corresponding to the reference facial image pair. By subtracting the image features of the two reference facial images in the reference facial image pair in the corresponding facial feature dimension, the first image feature corresponding to the reference facial image pair is obtained, so that the server can quickly analyze the difference between the two reference facial images in the reference facial image pair in the corresponding facial feature dimension based on the first image feature, thereby quickly determining the facial modification performed on the object in the reference facial image in the corresponding facial feature dimension.
[0123] In step S403, the server extracts features from the original facial image through the image editing module in the image processing model to obtain second image features.
[0124] In the disclosed embodiment, the image editing module includes an encoder for extracting features from the image. Accordingly, the server can extract features from the original facial image through the image editing module in the image processing model to obtain second image features. The second image features include multiple facial features of the object in the original facial image. The multiple facial features can be features such as skin color, facial features, and makeup, which are not limited in the disclosed embodiment.
[0125] It should be noted that the embodiment of the present disclosure does not limit the execution order of steps 401-402 and step 403. The embodiment of the present disclosure only takes the sequential execution of steps 401, 402 and 403 as an example for explanation; in other embodiments, the execution may be carried out in the order of step 403 first and then steps 401-402, or steps 401-402 and step 403 may be executed simultaneously.
[0126] In step S404, the server performs feature transformation on the multiple first image features based on the second image features through the feature processing module in the image processing model to obtain multiple third image features. The feature transformation is used to update the editing degree of the multiple first image features on the original facial image.
[0127] In the disclosed embodiment, when the original facial image is edited based on the reference facial image in the plurality of reference facial image pairs, if the facial features of the object in the result image obtained by editing are completely consistent with the facial features of the object in the reference facial image, then the result image will lack the facial features of the original facial image. To solve this problem, the electronic device can perform feature transformation on the plurality of first image features based on the second image features to update the editing degree of the plurality of first image features on the original facial image.
[0128] In some embodiments, when a user uploads multiple reference facial image pairs, the user can set the corresponding facial feature dimension for each reference facial image pair, that is, the user can determine the image feature of the reference facial image pair in a certain facial feature dimension as the image feature that affects the final generated result image. Further, the user can also set the weight of each facial feature dimension to change the degree of editing of the image feature of the reference facial image pair in the corresponding facial feature dimension on the original facial image, that is, to change the degree of influence of the image feature on the final generated result image. Accordingly, the server can receive an instruction to set the weights of multiple facial feature dimensions; for any facial feature dimension, in response to an instruction to set the weight of the facial feature dimension, the server can perform a feature transformation on the first image feature corresponding to the facial feature dimension based on the weight and the second image feature to obtain a third image feature. Wherein, when performing a feature transformation on the first image feature corresponding to the facial feature dimension based on the weight of the facial feature dimension and the second image feature, the server can first weight the first image feature corresponding to the facial feature dimension based on the weight of the facial feature dimension, and then perform a feature transformation on the weighted first image feature based on the second image feature to obtain the third image feature. By weighting the first image feature corresponding to each facial feature dimension based on the weights of multiple facial feature dimensions manually set by the user, the user can easily set by manual adjustment which facial feature dimension of the reference facial image in the reference facial image pair is used to influence the final generated result image, as well as the degree of influence of the image features of the reference facial image in the reference facial image pair in the corresponding facial feature dimension on the final generated result image, thereby achieving user-controllable editing of facial images.
[0129] For example, when a user imports multiple reference facial image pairs by triggering an image import control in an application, the terminal can display a setting interface in the application, and the setting interface displays multiple reference facial image pairs. For any reference facial image pair, in response to a dimension setting operation on the reference facial image pair, the terminal can display multiple facial feature dimensions for the user to select. In response to a user's selection operation on a certain facial feature dimension, the terminal can determine the facial feature dimension as the facial feature dimension in the reference facial image pair that affects the result image. Further, in response to a weight setting operation on the facial feature dimension corresponding to the reference facial image pair, the terminal can display a weight adjustment component. The user can change the weight of the facial feature dimension by sliding the weight adjustment component left and right, thereby changing the degree of influence of the image features of the reference facial image pair on the facial feature dimension on the result image. In response to a sliding operation on the weight adjustment component, the terminal can determine the weight of the facial feature dimension according to the sliding position of the weight adjustment component.
[0130] It should be noted that the embodiments of the present disclosure only take the above-mentioned interface operation form as an example for explanation, and the dimension setting operation and the weight setting operation may also be other interface operation forms, and the embodiments of the present disclosure are not limited to this.
[0131] In some embodiments, the server can perform feature transformation on the first image feature based on the similarity between the second image feature and the first image feature. Accordingly, for any first image feature, the server can determine the similarity corresponding to the first image feature based on the first image feature and the second image feature, and the similarity is used to indicate the degree of similarity between the first image feature and the second image feature; then, the server can perform feature transformation on the first image feature based on the similarity to obtain a third image feature. Among them, since the first image feature is used to indicate the difference between the first reference facial image and the second reference facial image in the corresponding facial feature dimension, when determining the similarity, the server can determine the facial features belonging to the facial feature dimension corresponding to the first image feature from the multiple facial features included in the second image feature, and then can determine the similarity corresponding to the first image feature based on the facial features belonging to the facial feature dimension in the second image feature and the first image feature. The similarity is used to indicate the degree of similarity between the facial features belonging to the facial feature dimension in the second image feature and the first image feature. By performing feature transformation on the first image feature based on the similarity corresponding to the first image feature, the feature processing module can update the editing degree of the first image feature on the original facial image based on the similarity between the facial features belonging to the specific facial feature dimension in the second image feature and the first image feature, so that the result image obtained based on the image features of the original facial image and the multiple third image features obtained by the feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial characteristics of the original facial image, thereby improving the beautification effect of the result image.
[0132] In some embodiments, the server can weight the first image feature based on the similarity corresponding to the first image feature to obtain the third image feature. Accordingly, the server can update the feature transformation weight of the first image feature based on the similarity. Since the feature transformation weight is used to indicate the degree of editing of the original facial image by the first image feature, the server can weight the first image feature based on the feature transformation weight to obtain the third image feature. By updating the feature transformation weight of the first image feature based on the similarity corresponding to the first image feature, the degree of editing of the original facial image by the first image feature can be changed as the degree of similarity between the first image feature and the second image feature changes, so that the result image obtained based on the image features of the original facial image and the multiple third image features obtained by the feature transformation can selectively retain some image features in the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.
[0133] In some embodiments, the server can increase or decrease the feature transformation weight of the first image feature based on the similarity corresponding to the first image feature. Accordingly, when the similarity corresponding to the first image feature is greater than the first threshold, it indicates that the similarity between the first image feature and the second image feature is high, that is, it indicates that the similarity between the facial features of the first image feature and the second image feature belonging to the facial feature dimension corresponding to the first image feature is high. Therefore, the server can determine that the facial features of the second image feature belonging to the facial feature dimension are high-quality facial features in the original facial image, and then reduce the feature transformation weight of the first image feature to reduce the editing degree of the first image feature to the original facial image. When the similarity corresponding to the first image feature is less than the second threshold, it indicates that the similarity between the first image feature and the second image feature is low, that is, it indicates that the similarity between the facial features of the first image feature and the second image feature belonging to the facial feature dimension corresponding to the first image feature is low. Therefore, the server can determine that the facial features of the second image feature belonging to the facial feature dimension are low-quality facial features in the original facial image, and then increase the feature transformation weight of the first image feature to increase the editing degree of the first image feature to the original facial image. Wherein, the second threshold is not greater than the first threshold. By reducing the feature transformation weight of the first image feature when the similarity is high, and increasing the feature change weight of the first image feature when the similarity is low, the result image obtained based on the second image feature and the third image feature obtained by feature transformation can retain some image features of the original facial image that have high similarity with the first image feature, that is, the result image can retain higher quality facial features in the original facial image, thereby improving the beautification effect of the result image.
[0134] In step S405, for any third image feature, the server determines the editing degree corresponding to the third image feature based on the third image feature and the second image feature through the image editing module in the image processing model, where the editing degree is used to indicate the degree of editing of the third image feature to the second image feature, and the editing degree is proportional to the difference between the third image feature and the second image feature.
[0135] In the disclosed embodiment, the image editing module also includes a decoder for decoding the image feature and outputting the image represented by the image feature. Accordingly, the server can decode the second image feature and the multiple third image features obtained by the feature transformation through the image editing module in the image processing model to obtain a result image. In the process of decoding the second image feature and the multiple third image features obtained by the feature transformation, the server can first determine the editing degree corresponding to the multiple third image features, and then decode the second image feature based on the editing degree to obtain the result image. Among them, the editing degree corresponding to the third image feature is used to indicate the degree of editing of the second image feature by the third image feature.
[0136] Since the third image feature is the first image feature after feature transformation, when determining the degree of editing of any third image feature on the second image feature, the server can determine the facial features belonging to the facial feature dimension from the multiple facial features included in the second image feature based on the facial feature dimension corresponding to the first image feature, and then determine the degree of editing corresponding to the third image feature based on the facial features belonging to the facial feature dimension in the second image feature and the third image feature.
[0137] In step S406, the server decodes the second image feature based on the editing degrees corresponding to the plurality of third image features through the image editing module in the image processing model to obtain a result image.
[0138] In the disclosed embodiment, the second image feature includes not only multiple facial features of the object in the original facial image, but also other image features other than facial features, such as background features of the original facial image, expression features of the object, and posture features, etc. Accordingly, in the process of decoding the second image feature based on the editing degrees corresponding to the multiple third image features, the server can first decode the other image features other than facial features in the second image feature through the image editing module to obtain other image parts other than the face of the object in the result image. Then the server can edit the multiple facial features in the second image feature based on the editing degrees corresponding to the multiple third image features to obtain the multiple facial features of the object in the result image. Finally, the server can decode the edited multiple facial features through the image editing module to obtain the face of the object in the result image. By decoding the second image feature based on the editing degrees corresponding to the multiple third image features obtained based on the special transformation, the server can edit the original facial image according to the editing degrees of the original facial image by the multiple third image features, so that the edited result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby improving the beautification effect of the result image.
[0139] In some embodiments, the server can decode the edited multiple facial features in sequence according to a specific order. Take the multiple facial features as skin color and texture features, makeup features, and facial features as an example. During the decoding process, the server can first decode the facial features to reconstruct the global structure of the object's face in the result image. Then, the server can decode the makeup features to reconstruct the global color of the object's face in the result image. Finally, the server can decode the skin color and texture features to reconstruct the facial details of the object's face in the result image. By decoding multiple facial features in sequence, the image editing module can reconstruct the result image in an orderly manner, thereby improving the generation efficiency of the result image, that is, improving the processing efficiency of the facial image.
[0140] In some embodiments, in order to ensure the accuracy of the image processing model, the server needs to train the image processing model. The training process of the image processing model includes the following steps:
[0141] (1) The server processes multiple sample reference facial image pairs and sample original facial images through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object. The sample objects included in the sample reference facial images in the sample reference facial image pairs may be the same as or different from the sample objects included in the sample original facial images, and the embodiments of the present disclosure are not limited to this. The prediction result image is an image obtained by editing the sample original images based on the sample reference facial images in the multiple sample reference facial image pairs, that is, the objects in the prediction result image are the same as the sample objects in the sample original facial images, and the prediction result image retains some image features of the sample original facial images while including the image features of the sample reference facial images.
[0142] In some embodiments, the process of the server processing multiple sample reference facial image pairs and sample original facial images through an image processing model includes: processing multiple sample reference facial image pairs through an image processing model to obtain multiple first sample image features; extracting features from the sample original images through an image processing model to obtain second sample image features; performing feature transformation on multiple first sample image features based on the second sample image features through an image processing model to obtain multiple third sample image features, wherein the feature transformation is used to update the editing degree of the first sample image features on the sample original facial image; decoding the second sample image features and multiple third sample image features through an image processing model to obtain a prediction result image. The first sample image features are used to represent the difference between the two sample reference facial images in the sample reference facial image pair in the corresponding facial feature dimensions, and different first sample image features correspond to different facial feature dimensions. The second sample image features are used to represent the sample original facial image. In order to prevent the facial features of the object in the prediction result image from being completely identical to those of the sample reference facial image, resulting in the prediction result image lacking the facial features of the sample original facial image. The server can perform feature transformation on multiple first sample image features based on the second sample image features to obtain multiple third sample image features. By performing feature transformation on multiple first sample image features based on second sample image features, the editing degree of the sample original facial image by the multiple first sample image features can be updated, and then when the image features of the sample original facial image and the multiple third sample image features obtained by the feature transformation are decoded, the decoded prediction result image can retain some image features of the sample original facial image, that is, the prediction result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the prediction result image output by the image processing model.
[0143] In some embodiments, the process of the server performing feature transformation on multiple first sample image features based on the second sample image feature through the image processing model includes: for any first sample image feature, determining the similarity corresponding to the first sample image feature based on the first sample image feature and the sample second image feature through the image processing model, the similarity is used to indicate the degree of similarity between the first sample image feature and the second sample image feature; based on the similarity, performing feature transformation on the first sample image feature to obtain a third sample image feature. Wherein, since the first sample image feature is used to indicate the difference between the two sample reference facial images in the sample reference facial image pair in the corresponding facial feature dimension, when determining the similarity, the server can determine the facial features belonging to the facial feature dimension corresponding to the first sample image feature from the multiple facial features included in the second sample image feature, and then can determine the similarity corresponding to the first sample image feature based on the facial features belonging to the facial feature dimension in the second sample image feature and the first sample image feature. The similarity is used to indicate the degree of similarity between the facial features belonging to the facial feature dimension in the second sample image feature and the first sample image feature. By performing feature transformation on the first sample image feature based on the similarity corresponding to the first sample image feature, the feature processing module can update the editing degree of the first sample image feature on the sample original facial image based on the similarity between the facial features belonging to the specific facial feature dimension in the second sample image feature and the first sample image feature, so that the predicted result image obtained based on the image features of the sample original facial image and multiple third sample image features obtained by feature transformation can selectively retain some sample image features in the sample original facial image, that is, the predicted result image can have the facial features of the sample original facial image, thereby improving the beautification effect of the predicted result image.
[0144] (2) Since the multiple facial features in the prediction result image must be consistent with the multiple third sample image features corresponding to the multiple sample reference facial image pairs, for any sample reference facial image pair, the server can determine the first training loss based on the sample reference facial image pair and the prediction result image. The first training loss is used to indicate the difference between the sample reference facial image pair and the prediction result image, that is, to indicate the difference between the third sample image feature corresponding to the sample reference facial image pair and the image feature of the prediction result image. Since the third sample image feature is the first sample image feature after feature transformation, when determining the first training loss, the server can determine the facial features belonging to the facial feature dimension from the image feature of the prediction result image based on the facial feature dimension corresponding to the first sample image feature, and then determine the first training loss based on the facial feature and the third sample image feature. The first training loss is used to indicate the difference between the third sample image feature and the facial features belonging to the facial feature dimension in the prediction result image. The facial feature dimension can be any one of the dimensions such as skin color, makeup, facial features and face shape. Accordingly, the first training loss can be any one of the skin color loss, makeup loss, and facial features and face shape loss. Skin color and texture loss is used to indicate the difference in skin color and texture between the sample reference facial image and the predicted result image, facial feature shape loss is used to indicate the difference in facial feature shape between the sample reference facial image and the predicted result image, and makeup loss is used to indicate the difference in makeup between the sample reference facial image and the predicted result image.
[0145] (3) Since other image features in the predicted result image except facial features must be consistent with the sample original facial image, the server can determine the second training loss based on the sample original facial image and the predicted result image. The second training loss is used to indicate the difference between the sample original facial image and the predicted result image, that is, to indicate the difference between other image features in the predicted result image and other image features in the sample original facial image. Other image features include background features of the image, expression features of the object in the image, posture features, etc. Accordingly, the second training loss may include background consistency loss, expression loss, posture loss, etc. Background consistency loss is used to indicate the difference in image background between the sample original facial image and the predicted result image, expression loss is used to indicate the difference in object expression between the sample original facial image and the predicted result image, and posture loss is used to indicate the difference in object posture between the sample original facial image and the predicted result image.
[0146] (4) The server trains the image processing model based on the multiple first training losses and the second training losses. The accuracy of the image processing model can be reflected by whether the image features of the predicted result image are consistent with the image features corresponding to the multiple sample reference facial images and the image features of the sample original images. Therefore, the multiple first training losses and the second training losses can be used to train the image processing model. By training the image processing model based on the multiple first training losses and the second training losses, the image features in the result image output by the trained image processing model can be consistent not only with the reference facial image, but also with the original facial image. In addition, the result image can include the image features of the reference facial image while retaining the image features of the original facial image, thereby improving the accuracy of the image processing model.
[0147] The present disclosure provides a facial image processing method, which can update the editing degree of the original facial image by the multiple first image features corresponding to the multiple reference facial image pairs by performing feature transformation on the multiple first image features corresponding to the multiple reference facial image pairs based on the image features of the original facial image, and then when decoding the image features of the original facial image and the multiple third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, so that the beautification effect of the processed facial image can be improved while ensuring the processing efficiency of the facial image.
[0148] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.
[0149] Figure 5 is a block diagram of a facial image processing device according to an exemplary embodiment. Figure 5 The device includes: a processing unit 501, a feature extraction unit 502, a feature transformation unit 503 and a decoding unit 504.
[0150] The processing unit 501 is configured to process a plurality of reference facial image pairs to obtain a plurality of first image features, each reference facial image pair including two reference facial images from the same subject, the first image features being used to represent a difference between the two reference facial images in the reference facial image pair in a corresponding facial feature dimension, and different first image features corresponding to different facial feature dimensions;
[0151] The feature extraction unit 502 is configured to extract features from the original facial image to obtain second image features;
[0152] A feature transformation unit 503 is configured to perform feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image;
[0153] The decoding unit 504 is configured to decode the second image feature and the plurality of third image features to obtain a result image.
[0154] In some embodiments, the reference facial image pair includes a first reference facial image and a second reference facial image, the first reference facial image being a reference facial image in which facial modification has not been performed on the subject, and the second reference facial image being a reference facial image in which facial modification has been performed on the subject;
[0155] The processing unit 501 includes:
[0156] The extraction subunit 5011 is configured to, for any reference facial image pair, perform feature extraction on the first reference facial image and the second reference facial image respectively to obtain a fourth image feature and a fifth image feature, wherein the fourth image feature is an image feature of the first reference facial image in a corresponding facial feature dimension, and the fifth image feature is an image feature of the second reference facial image in a corresponding facial feature dimension;
[0157] The processing subunit 5012 is configured to subtract the fourth image feature from the fifth image feature to obtain a first image feature corresponding to the reference facial image pair.
[0158] In some embodiments, the feature transformation unit 503 is also configured to receive instructions for setting weights for multiple facial feature dimensions; for any facial feature dimension, in response to instructions for setting weights for the facial feature dimension, a feature transformation is performed on the first image feature corresponding to the facial feature dimension based on the weight and the second image feature to obtain a third image feature.
[0159] In some embodiments, the feature conversion unit 503 includes:
[0160] The determination subunit 5031 is configured to determine, for any first image feature, a similarity corresponding to the first image feature based on the first image feature and the second image feature, where the similarity is used to indicate a degree of similarity between the first image feature and the second image feature;
[0161] The transformation subunit 5032 is configured to perform feature transformation on the first image feature based on the similarity to obtain a third image feature.
[0162] In some embodiments, the transformation subunit 5032 is configured to update the feature transformation weight of the first image feature based on the similarity, and the feature transformation weight is used to indicate the degree of editing of the original facial image by the first image feature; based on the feature transformation weight, the first image feature is weighted to obtain a third image feature.
[0163] In some embodiments, the transformation subunit 5032 is configured to reduce the feature transformation weight when the similarity is greater than a first threshold; and increase the feature transformation weight when the similarity is less than a second threshold, and the second threshold is not greater than the first threshold.
[0164] In some embodiments, the decoding unit 504 is configured to determine, for any third image feature, a degree of editing corresponding to the third image feature based on the third image feature and the second image feature, where the degree of editing is used to indicate the degree of editing of the second image feature by the third image feature, and the degree of editing is proportional to the difference between the third image feature and the second image feature; based on the editing degrees corresponding to multiple third image features, the second image feature is decoded to obtain a result image.
[0165] The present disclosure provides a facial image processing device, which can update the editing degree of the original facial image by the multiple first image features corresponding to the multiple reference facial image pairs by performing feature transformation on the multiple first image features corresponding to the multiple reference facial image pairs based on the image features of the original facial image, and then when decoding the image features of the original facial image and the multiple third image features obtained by the feature transformation, the decoded result image can retain some image features of the original facial image, that is, the result image can have the facial features of the original facial image, thereby ensuring the processing efficiency of the facial image while improving the beautification effect of the processed facial image.
[0166] It should be noted that the facial image processing device provided in the above embodiment only uses the division of the above functional units as an example to illustrate when processing facial images. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the facial image processing device provided in the above embodiment and the facial image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0167] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0168] Figure 6 is a block diagram of a model training device according to an exemplary embodiment. Figure 6 The device includes: a processing unit 601, a first determination unit 602, a second determination unit 603 and a training unit 604.
[0169] The processing unit 601 is configured to process a plurality of sample reference facial image pairs and a sample original facial image through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object;
[0170] A first determining unit 602 is configured to determine, for any sample reference facial image pair, a first training loss based on the sample reference facial image pair and the prediction result image, wherein the first training loss is used to indicate a difference between the sample reference facial image pair and the prediction result image;
[0171] A second determining unit 603 is configured to determine a second training loss based on the sample original facial image and the prediction result image, where the second training loss is used to indicate a difference between the sample original facial image and the prediction result image;
[0172] The training unit 604 is configured to train the image processing model based on multiple first training losses and second training losses.
[0173] In some embodiments, the processing unit 604 is configured to process multiple sample reference facial image pairs through an image processing model to obtain multiple first sample image features, where the first sample image features are used to represent the difference between two sample reference facial images in the sample reference facial image pair in corresponding facial feature dimensions, and different first sample image features correspond to different facial feature dimensions; to extract features from the sample original image through the image processing model to obtain second sample image features; to perform feature transformation on the multiple first sample image features based on the second sample image features through the image processing model to obtain multiple third sample image features, where the feature transformation is used to update the degree of editing of the sample original facial image by the first sample image features; and to decode the second sample image features and multiple third sample image features through the image processing model to obtain a predicted result image.
[0174] In some embodiments, the processing unit 604 is configured to determine, for any first sample image feature, the similarity corresponding to the first sample image feature based on the first sample image feature and the sample second image feature through an image processing model, where the similarity is used to indicate the degree of similarity between the first sample image feature and the second sample image feature; based on the similarity, perform a feature transformation on the first sample image feature to obtain a third sample image feature.
[0175] The present disclosure implements and provides a model training device, which trains an image processing model based on multiple first training losses and second training losses, so that the image features in the result image output by the trained image processing model can be consistent not only with the reference facial image, but also with the original facial image, and thus the result image can include the image features of the reference facial image while retaining the image features of the original facial image, thereby improving the accuracy of the image processing model.
[0176] It should be noted that the model training device provided in the above embodiment only uses the division of the above functional units as an example when training the image processing model. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the model training device provided in the above embodiment belongs to the same concept as the model training method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.
[0177] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0178] When an electronic device is provided as a terminal, Figure 7 FIG. 1 is a block diagram of a terminal according to an exemplary embodiment. Figure 7 As shown, the terminal 700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a notebook computer or a desktop computer. The terminal 700 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.
[0179] Typically, the terminal 700 includes a processor 701 and a memory 702 .
[0180] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0181] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the facial image processing method or model training method provided in the method embodiment of the present disclosure.
[0182] In some embodiments, the terminal 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709.
[0183] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.
[0184] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.
[0185] The display screen 705 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, and the front panel of the terminal 700 is set; in other embodiments, the display screen 705 can be at least two, which are respectively set on different surfaces of the terminal 700 or are folded; in some other embodiments, the display screen 705 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).
[0186] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0187] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.
[0188] The power supply 708 is used to power various components in the terminal 700. The power supply 708 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 708 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0189] In some embodiments, the terminal 700 further includes one or more sensors 709 , including but not limited to: an acceleration sensor 7010 , a gyroscope sensor 7011 , a pressure sensor 7012 , an optical sensor 7013 , and a proximity sensor 7014 .
[0190] The acceleration sensor 7010 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 700. For example, the acceleration sensor 7010 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 701 can control the display screen 705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 7010. The acceleration sensor 7010 can also be used for collecting game or user motion data.
[0191] The gyro sensor 7011 can detect the body direction and rotation angle of the terminal 700, and the gyro sensor 7011 can cooperate with the acceleration sensor 7010 to collect the user's 3D actions on the terminal 700. The processor 701 can implement the following functions based on the data collected by the gyro sensor 7011: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0192] The pressure sensor 7012 can be set on the side frame of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 7012 is set on the side frame of the terminal 700, it can detect the user's holding signal of the terminal 700, and the processor 701 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 7012. When the pressure sensor 7012 is set on the lower layer of the display screen 705, the processor 701 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0193] The optical sensor 7013 is used to collect the ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 according to the ambient light intensity collected by the optical sensor 7013. Optionally, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is reduced. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 7013.
[0194] The proximity sensor 7014, also called a distance sensor, is disposed on the front panel of the terminal 700. The proximity sensor 7014 is used to collect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 7014 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from the screen-on state to the screen-off state; when the proximity sensor 7014 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from the screen-off state to the screen-on state.
[0195] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 700, and the terminal 700 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0196] When the electronic device is provided as a server, Figure 8It is a block diagram of a server according to an exemplary embodiment. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one program code, and the at least one program code is loaded and executed by the processor 801 to implement the facial image processing method or model training method provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 800 may also include other components for implementing device functions, which will not be described in detail here.
[0197] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 702 or a memory 802 including instructions, and the instructions can be executed by the processor 701 of the terminal 700 or the processor 801 of the server 800 to complete the above-mentioned facial image processing method, or the above-mentioned model training method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0198] A computer program product includes a computer program, which implements the above-mentioned facial image processing method or the above-mentioned model training method when executed by a processor.
[0199] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0200] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A facial image processing method, characterized in that: The method comprises: Processing a plurality of reference facial image pairs to obtain a plurality of first image features, each reference facial image pair comprising two reference facial images from the same subject, the first image features being used to represent a difference between the two reference facial images in the reference facial image pair in a corresponding facial feature dimension, and different first image features corresponding to different facial feature dimensions; Extracting features from the original facial image to obtain second image features; Based on the second image feature, performing feature transformation on the plurality of first image features to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image; The second image feature and the plurality of third image features are decoded to obtain a result image.
2. The facial image processing method according to claim 1, characterized in that: The reference facial image pair includes a first reference facial image and a second reference facial image, wherein the first reference facial image is a reference facial image in which facial modification is not performed on the subject, and the second reference facial image is a reference facial image in which facial modification is performed on the subject; The processing of the plurality of reference facial image pairs to obtain a plurality of first image features includes: For any reference facial image pair, performing feature extraction on the first reference facial image and the second reference facial image respectively to obtain a fourth image feature and a fifth image feature, wherein the fourth image feature is an image feature of the first reference facial image in a corresponding facial feature dimension, and the fifth image feature is an image feature of the second reference facial image in a corresponding facial feature dimension; The fourth image feature is subtracted from the fifth image feature to obtain a first image feature corresponding to the reference facial image pair.
3. The facial image processing method according to claim 1, characterized in that: The method further comprises: receiving an instruction for setting weights of a plurality of facial feature dimensions; The step of performing feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features includes: For any facial feature dimension, in response to an instruction to set a weight for the facial feature dimension, a feature transformation is performed on a first image feature corresponding to the facial feature dimension based on the weight and the second image feature to obtain a third image feature.
4. The facial image processing method according to claim 1, characterized in that: The step of performing feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features includes: For any first image feature, determining a similarity corresponding to the first image feature based on the first image feature and the second image feature, where the similarity is used to indicate a degree of similarity between the first image feature and the second image feature; Based on the similarity, feature transformation is performed on the first image feature to obtain the third image feature.
5. The facial image processing method according to claim 4, characterized in that: The performing feature transformation on the first image feature based on the similarity to obtain the third image feature includes: Based on the similarity, updating a feature transformation weight of the first image feature, the feature transformation weight being used to indicate a degree of editing of the original facial image by the first image feature; The first image feature is weighted based on the feature transformation weight to obtain the third image feature.
6. The facial image processing method according to claim 5, characterized in that: The updating of the feature transformation weight of the first image feature based on the similarity includes: When the similarity is greater than a first threshold, reducing the feature transformation weight; When the similarity is less than a second threshold, the feature transformation weight is increased, and the second threshold is not greater than the first threshold.
7. The facial image processing method according to claim 1, characterized in that: The decoding of the second image feature and the plurality of third image features to obtain a result image includes: For any third image feature, determining, based on the third image feature and the second image feature, an editing degree corresponding to the third image feature, the editing degree being used to indicate the degree of editing of the second image feature by the third image feature, and the editing degree being proportional to the difference between the third image feature and the second image feature; Based on the editing degrees corresponding to the plurality of third image features, the second image feature is decoded to obtain the result image.
8. A model training method, characterized in that: The method comprises: Processing a plurality of sample reference facial image pairs and sample original facial images through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object; For any sample reference facial image pair, determining a first training loss based on the sample reference facial image pair and the prediction result image, wherein the first training loss is used to indicate a difference between the sample reference facial image pair and the prediction result image; Determining a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image; The image processing model is trained based on multiple first training losses and the second training losses.
9. The model training method according to claim 8, characterized in that: The method of processing a plurality of sample reference facial image pairs and a sample original facial image through an image processing model to obtain a prediction result image includes: The plurality of sample reference facial image pairs are processed by the image processing model to obtain a plurality of first sample image features, wherein the first sample image features are used to represent the difference between two sample reference facial images in the sample reference facial image pairs in corresponding facial feature dimensions, and different first sample image features correspond to different facial feature dimensions; By using the image processing model, feature extraction is performed on the sample original image to obtain second sample image features; By means of the image processing model, based on the second sample image feature, the plurality of first sample image features are subjected to feature transformation to obtain a plurality of third sample image features, wherein the feature transformation is used to update the editing degree of the first sample image feature on the sample original facial image; The second sample image feature and the plurality of third sample image features are decoded by the image processing model to obtain the prediction result image.
10. The model training method according to claim 9, characterized in that: The step of performing feature transformation on the plurality of first sample image features based on the second sample image features through the image processing model to obtain a plurality of third sample image features includes: For any first sample image feature, determining, by means of the image processing model, a similarity corresponding to the first sample image feature based on the first sample image feature and the sample second image feature, wherein the similarity is used to indicate a degree of similarity between the first sample image feature and the second sample image feature; Based on the similarity, feature transformation is performed on the first sample image feature to obtain the third sample image feature.
11. A facial image processing device, characterized in that: The device comprises: a processing unit configured to process a plurality of reference facial image pairs to obtain a plurality of first image features, each reference facial image pair comprising two reference facial images from the same subject, the first image features being used to represent a difference between the two reference facial images in the reference facial image pair in a corresponding facial feature dimension, and different first image features corresponding to different facial feature dimensions; A feature extraction unit is configured to extract features from the original facial image to obtain second image features; a feature transformation unit configured to perform feature transformation on the plurality of first image features based on the second image feature to obtain a plurality of third image features, wherein the feature transformation is used to update the editing degree of the plurality of first image features on the original facial image; A decoding unit is configured to decode the second image feature and the plurality of third image features to obtain a result image.
12. A model training device, characterized in that: The device comprises: A processing unit is configured to process a plurality of sample reference facial image pairs and a sample original facial image through an image processing model to obtain a prediction result image, wherein each sample reference facial image pair includes two sample reference facial images from the same sample object; A first determining unit is configured to determine, for any sample reference facial image pair, a first training loss based on the sample reference facial image pair and the prediction result image, wherein the first training loss is used to indicate a difference between the sample reference facial image pair and the prediction result image; A second determining unit is configured to determine a second training loss based on the sample original facial image and the prediction result image, wherein the second training loss is used to indicate a difference between the sample original facial image and the prediction result image; A training unit is configured to train the image processing model based on a plurality of the first training losses and the second training losses.
13. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; Wherein, the processor is configured to execute the program code to implement the facial image processing method as described in any one of claims 1 to 7, or the model training method as described in any one of claims 8 to 10.
14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the facial image processing method as described in any one of claims 1 to 7, or the model training method as described in any one of claims 8 to 10.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the facial image processing method described in any one of claims 1 to 7, or the model training method described in any one of claims 8 to 10.
Citation Information
Cited By
Image retouching method and electronic device
WO2026170833A1
