Facial expression adjustment model training method, facial expression adjustment method, and related device

Abstract

The present disclosure relates to a training method of an expression adjustment model, an expression adjustment method and related equipment. The training method of the expression adjustment model comprises: obtaining a first image and a second image corresponding to the same object; an expression coefficient of the object in the first image is greater than a first preset value, and an expression coefficient of the object in the second image is less than or equal to the first preset value; a first target region is cropped from the first image to obtain a third image; a second target region is occluded in the first image to obtain a fourth image; the first target region contains the second target region; the third image and the fourth image are input into a first preset model to obtain a fifth image different from the expression coefficient of the first image; a first loss value is determined based on the second image and the fifth image, and a model parameter of the first preset model is adjusted based on the first loss value to obtain an expression adjustment model. The expression adjustment model has a good effect when used for expression adjustment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the image processing technology, and particularly relates to a training method of an expression adjustment model, an expression adjustment method and related equipment. BACKGROUND

[0002] Facial expression plays an important role in daily life and communication, but there are some application scenarios in which images with expressions will have an adverse effect on the final result. For example, for a live photo effect, when the input user image is in a natural expressionless state with eyes open and mouth closed, the live photo effect is good, and when the input user image is in an expression state such as pouting, pursing the lips, or opening the mouth, the live photo effect is poor. In order to reduce the adverse effect of images with expressions, expression adjustment needs to be performed.

[0003] In related technologies, when expression editing is performed based on a PAGAN (a kind of neural network model), expression adjustment can be performed by controlling expression coefficients, Figure 1 The left image shows an input image of pouting, and the right image shows an output image after expression adjustment. As can be seen from the comparison, the adjustment effect is poor when expression adjustment is performed in related technologies. SUMMARY

[0004] The present disclosure provides a training method of an expression adjustment model, an expression adjustment method and related equipment to at least solve the technical problem of poor adjustment effect when expression adjustment is performed in related technologies. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a training method of an expression adjustment model is provided, comprising:

[0006] obtaining a first image and a second image corresponding to the same object; the expression coefficient of the object in the first image is greater than a first preset value, and the expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to represent the facial expression amplitude in the image; and the first preset value is a threshold value preset for binary classification of the facial expression amplitude;

[0007] cropping a first target region in the first image to obtain a third image; the first target region is a local facial region containing a pre-set facial organ and embodying the identity feature of the object;

[0008] performing occlusion processing on a second target region in the first image to obtain a fourth image; the second target region is a facial organ region corresponding to the facial organ; and the first target region contains the second target region;

[0009] inputting the third image and the fourth image into a first preset model to obtain a fifth image with a different expression coefficient from the first image;

[0010] determining a first loss value based on the second image and the fifth image, adjusting a model parameter of the first preset model based on the first loss value to obtain an expression adjustment model.

[0011] Optionally, the occlusion processing on the second target region in the first image to obtain a fourth image comprises:

[0012] In a case where the expression coefficient corresponding to the first image is less than or equal to a second preset value, adjusting a pixel value in the second target region in the first image to a third preset value set in advance for pixel occlusion to perform occlusion processing on the second target region to obtain a fourth image; the second preset value is greater than the first preset value.

[0013] Optionally, the method further comprises:

[0014] In a case where the expression coefficient corresponding to the first image is greater than the second preset value, performing animation processing on the first image to obtain a new first image; the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value;

[0015] adjusting a pixel value in the second target region in the new first image to the third preset value to perform occlusion processing on the second target region to obtain a fourth image.

[0016] Optionally, the animation processing on the first image to obtain a new first image comprises:

[0017] performing three-dimensional reconstruction on the first image to determine a three-dimensional face model;

[0018] adjusting an expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determining a new first image based on the three-dimensional face model with the adjusted expression coefficient.

[0019] Optionally, the first preset model comprises a first neural network, a second neural network and a third neural network; and the inputting the third image and the fourth image into the first preset model to obtain a fifth image with a different expression coefficient from the first image comprises:

[0020] inputting the third image into the first neural network to perform feature extraction to obtain identity feature information in the third image, the identity feature information being used to distinguish different objects;

[0021] inputting the fourth image into the second neural network to perform feature extraction to obtain face attribute information of the object outside the face organ, the face attribute information being used to reflect a face feature in the fourth image.

[0022] inputting the identity feature information and the face attribute information into the third neural network for feature fusion to obtain a fifth image different from the expression coefficient of the first image.

[0023] Optionally, the method further includes:

[0024] obtaining a face image;

[0025] inputting the face image into an expression editing model to obtain an image sequence of expression changes;

[0026] In the image sequence, a first image with an expression coefficient greater than the first preset value and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value are selected.

[0027] Optionally, before the step of inputting the face image into the expression editing model to obtain the image sequence of expression changes, the method further includes:

[0028] obtaining a sixth image, the sixth image being training data of the expression editing model;

[0029] inputting the sixth image into a face expression recognition model to determine an expression coefficient corresponding to the sixth image;

[0030] inputting the expression coefficient corresponding to the sixth image and random noise into a second preset model to obtain a seventh image different from the expression coefficient of the sixth image;

[0031] determining a second loss value based on the sixth image and the seventh image, adjusting model parameters of the second preset model based on the second loss value to obtain the expression editing model.

[0032] Optionally, before the step of determining the second loss value based on the sixth image and the seventh image, adjusting the model parameters of the second preset model based on the second loss value, and obtaining the expression editing model, the method further includes:

[0033] inputting the seventh image into the face expression recognition model to obtain an expression coefficient corresponding to the seventh image;

[0034] determining a first sub-loss value based on a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image;

[0035] determining a second sub-loss value based on an average loss corresponding to the sixth image and the seventh image;

[0036] A determination result that the sixth image and the seventh image correspond to the same object is determined as a third sub-loss value;

[0037] A calculation result of the first sub-loss value, the second sub-loss value, and the third sub-loss value is determined as a second loss value of the sixth image and the seventh image.

[0038] According to a second aspect of the embodiments of the present disclosure, an expression adjustment method is provided, comprising:

[0039] Obtaining a to-be-processed image;

[0040] In a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, a first target region is cropped from the to-be-processed image to obtain a cropped image, and a second target region is subjected to occlusion processing in the to-be-processed image to obtain a to-be-processed image in which the second target region is occluded; the first target region is a local face region of a pre-set face organ and embodies an identity feature of an object, and the second target region is a face organ region corresponding to the face organ; the first target region contains the second target region.

[0041] The cropped image and the to-be-processed image in which the second target region is occluded are input into an expression adjustment model to obtain a target image; an expression coefficient of the target image is less than or equal to the preset value.

[0042] The expression adjustment model is obtained according to the training method of the expression adjustment model of the first aspect.

[0043] According to a third aspect of the embodiments of the present disclosure, a training device of an expression adjustment model is provided, comprising:

[0044] An image acquisition module is configured to acquire a first image and a second image corresponding to the same object; an expression coefficient of the object in the first image is greater than a first preset value, and an expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to represent a face expression amplitude in an image; the first preset value is a threshold value pre-set for binary classification of a face expression amplitude;

[0045] A cropping processing module is configured to crop a first target region from the first image to obtain a third image; the first target region includes a local face region of a pre-set face organ and embodies an identity feature of the object;

[0046] An occlusion processing module is configured to perform occlusion processing on a second target region in the first image to obtain a fourth image; the second target region is a face organ region corresponding to the face organ; the first target region includes the second target region.

[0047] an image generation module configured to input the third image and the fourth image into a first preset model to obtain a fifth image different from the expression coefficient of the first image;

[0048] a parameter adjustment module configured to determine a first loss value based on the second image and the fifth image, and adjust a model parameter of the first preset model based on the first loss value to obtain an expression adjustment model.

[0049] Optionally, the occlusion processing module comprises:

[0050] a first occlusion processing unit configured to, in a case where the expression coefficient corresponding to the first image is less than or equal to a second preset value, adjust a pixel value in a second target region in the first image to a third preset value preset for pixel occlusion to perform occlusion processing on the second target region to obtain a fourth image; the second preset value is greater than the first preset value.

[0051] Optionally, the occlusion processing module further comprises:

[0052] a second occlusion processing unit configured to, in a case where the expression coefficient corresponding to the first image is greater than the second preset value, perform animation processing on the first image to obtain a new first image; the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value; and adjust a pixel value in a second target region in the new first image to the third preset value to perform occlusion processing on the second target region to obtain a fourth image.

[0053] Optionally, the second occlusion processing unit comprises:

[0054] a reconstruction processing sub-unit configured to perform three-dimensional reconstruction on the first image to determine a three-dimensional face model;

[0055] an image determination sub-unit configured to adjust an expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determine a new first image based on the three-dimensional face model after adjustment of the expression coefficient.

[0056] Optionally, the image generation module comprises:

[0057] a first feature extraction unit configured to input the third image into the first neural network to perform feature extraction to obtain identity feature information in the third image, the identity feature information being used to distinguish different objects;

[0058] The second feature extraction unit is configured to input the fourth image into the second neural network for feature extraction, to obtain face attribute information of the object outside the face organ, the face attribute information being used to reflect face features in the fourth image.

[0059] The feature fusion processing unit is configured to input the identity feature information and the face attribute information into the third neural network for feature fusion, to obtain a fifth image different from the expression coefficient of the first image.

[0060] Optionally, the image acquisition module comprises:

[0061] The image acquisition unit is configured to acquire a face image.

[0062] The image editing unit is configured to input the face image into an expression editing model, to acquire an image sequence of expression changes.

[0063] The image selection unit is configured to select, from the image sequence, a first image with an expression coefficient greater than the first preset value, and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value.

[0064] Optionally, the apparatus further comprises an editing model determination module, the editing model determination module comprising:

[0065] The training data acquisition unit is configured to acquire a sixth image, the sixth image being training data of the expression editing model.

[0066] The expression coefficient determination unit is configured to input the sixth image into a face expression recognition model, to determine an expression coefficient corresponding to the sixth image.

[0067] The expression coefficient adjustment unit is configured to input the expression coefficient corresponding to the sixth image and random noise into a second preset model, to obtain a seventh image different from the expression coefficient of the sixth image.

[0068] The editing model determination unit is configured to determine a second loss value based on the sixth image and the seventh image, to adjust model parameters of the second preset model based on the second loss value, and to obtain the expression editing model.

[0069] Optionally, the editing model determination module further comprises a loss value determination unit configured to input the seventh image into the human face expression recognition model to obtain an expression coefficient corresponding to the seventh image; determine a first sub-loss value as a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image; determine a second sub-loss value as an average loss corresponding to the sixth image and the seventh image; determine a third sub-loss value as a determination result of the same object corresponding to the sixth image and the seventh image; and determine a second loss value of the sixth image and the seventh image as a calculation result of the first sub-loss value, the second sub-loss value and the third sub-loss value.

[0070] According to a fourth aspect of the embodiments of the present disclosure, an expression adjustment apparatus is provided, comprising:

[0071] an image acquisition module configured to acquire a to-be-processed image;

[0072] an image processing module configured to, in a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, crop a first target region from the to-be-processed image to obtain a cropped image, and perform occlusion processing on a second target region in the to-be-processed image to obtain a to-be-processed image in which the second target region is occluded; the first target region is a local face region of a pre-set human face organ and embodies an identity feature of an object, and the second target region is a human face organ region corresponding to the human face organ; and the first target region contains the second target region;

[0073] a model processing module configured to input the cropped image and the to-be-processed image in which the second target region is occluded into an expression adjustment model to obtain a target image; an expression coefficient of the target image is less than or equal to the preset value;

[0074] wherein the expression adjustment model is obtained according to the training method of the expression adjustment model.

[0075] According to a fifth aspect of the embodiments of the present disclosure, an electronic device is provided, comprising:

[0076] a processor;

[0077] a memory for storing instructions executable by the processor;

[0078] wherein the processor is configured to execute the instructions to implement the training method of the expression adjustment model according to the first aspect, or implement the expression adjustment method according to the second aspect.

[0079] According to a sixth aspect of the embodiments of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the expression adjustment model according to the first aspect, or implement the expression adjustment method according to the second aspect.

[0080] According to a seventh aspect of the embodiments of the present disclosure, a computer program product is provided. The computer program product includes readable program codes. When the readable program codes are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the expression adjustment model according to the first aspect, or implement the expression adjustment method according to the second aspect.

[0081] The embodiments of the present disclosure provide at least the following beneficial effects:

[0082] The present disclosure accurately classifies the facial expression amplitude by using the first preset value, and then accurately obtains the first image and the second image belonging to different classifications, wherein the expression coefficient of the object in the first image is greater than the first preset value, and the expression coefficient of the object in the second image is less than or equal to the first preset value. The first target region containing the pre-set facial organ and embodying the identity feature of the object is determined in the first image, so as to crop the first target region in the first image to obtain a third image. The third image can effectively embody the identity feature of the object. The second target region corresponding to the facial organ is determined in the first image, wherein the first target region includes the second target region. The second target region is subjected to occlusion processing in the first image to obtain a fourth image. The facial expression amplitude is usually expressed by the facial organ. The second target region is subjected to occlusion processing to effectively remove the expression feature expressed by the facial organ in the second target region in the fourth image. Further, the third image and the fourth image are input into the first preset model to obtain a fifth image different from the expression coefficient of the first image. The second image with the expression coefficient less than or equal to the first preset value is taken as a learning target. The first loss value corresponding to the second image and the fifth image is determined. According to the first loss value, the model parameters of the first preset model are adjusted to obtain an expression adjustment model. The expression adjustment model provides the identity feature of the object through the third image, and effectively removes the expression feature expressed by the facial organ in the second target region, reduces the influence of the expression feature expressed by the facial organ in the second target region on the fifth image, avoids the possibility that the expression coefficient of the obtained fifth image is still large, and further ensures that the expression adjustment model can effectively reduce the expression coefficient of the object when performing expression adjustment, and obtain a better adjustment effect.

[0083] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0084] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure and, without in any way intending to unfairly limit the present disclosure, do not constitute an improper limitation thereof.

[0085] Figure 1 is an expression adjustment effect schematic diagram;

[0086] Figure 2 is a flowchart of a training method of an expression adjustment model according to an example embodiment;

[0087] Figure 3 is a schematic diagram of a first image and a second image in a training method of an expression adjustment model according to an example embodiment;

[0088] Figure 4 is a schematic diagram of an expression editing model in a training method of an expression adjustment model according to an example embodiment;

[0089] Figure 5 is a schematic diagram of an expression adjustment model in a training method of an expression adjustment model according to an example embodiment;

[0090] Figure 6 is an expression adjustment effect schematic diagram in a training method of an expression adjustment model according to an example embodiment;

[0091] Figure 7 is an animation processing schematic diagram in a training method of an expression adjustment model according to an example embodiment;

[0092] Figure 8 is a flowchart of an expression adjustment method according to an example embodiment;

[0093] Figure 9 is a block diagram of a training device of an expression adjustment model according to an example embodiment;

[0094] Figure 10 is a block diagram of an expression adjustment device according to an example embodiment;

[0095] Figure 11 is a block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0096] In order to make ordinary people in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0097] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0098] Figure 2 This describes a training method for an expression adjustment model according to an exemplary embodiment. This training method can be applied to a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The method specifically includes the following steps:

[0099] In step S21, a first image and a second image corresponding to the same object are obtained; the expression coefficient of the object in the first image is greater than a first preset value, and the expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to characterize the facial expression amplitude in the image; the first preset value is a pre-set threshold for binary classification of facial expression amplitude.

[0100] In this step, a first preset value is set in advance. This first preset value is a threshold for the expression coefficient, used to perform binary classification of facial expression amplitude. Specifically, an expression coefficient greater than the first preset value can be considered an image with expression, while an expression coefficient less than or equal to the first preset value can be considered an image without expression. The user can set the first preset value according to the actual scenario. A first image and a second image are acquired. The first image and the second image correspond to the same person. If the expression coefficient of the object in the first image is greater than the first preset value, then the first image is an image with expression, such as a pouting image, a pursed-lip image, or something similar. Figure 3 The image on the left shows an open mouth ( Figure 3 (The eye state, not shown in the left image, is the normal open eye state); the expression coefficient of the object in the second image is less than or equal to the first preset value, that is, the second image is an expressionless image of the same object as the first image, such as... Figure 3 The image on the right shows a closed-mouth expression that is close to a natural, expressionless state. Figure 3 (The eye state, not shown in the right image, is the normal open eye state). The first and second images are both training data used for model training.

[0101] In an embodiment, the step S21 of acquiring the first image and the second image corresponding to the same object comprises:

[0102] In step S211, a face image is acquired.

[0103] In this step, an image containing a face is acquired, and the expression coefficient of the face image can be any value, i.e., the face image can be an expression image or a non-expression image.

[0104] In step S212, the face image is input into an expression editing model to acquire an image sequence of expression changes.

[0105] In this step, the expression editing model is a pre-trained data model meeting a preset accuracy, which is used to edit the expression of a given face image to acquire a new image with a different expression, and determine an image sequence of expression changes based on the images with different expressions, i.e., there are expression images and non-expression images with different expression coefficients in the image sequence.

[0106] For example, the face image shown in the right image of FIG. 1 is input into the expression editing model to output the face image shown in the left image of FIG. 2; or the face image shown in the left image of FIG. 1 is input into the expression editing model to output the face image shown in the right image of FIG. 2. Of course, more images belonging to the object can be acquired to further acquire an image sequence of expression changes. Figure 3 Figure 3 Figure 3 Figure 3

[0107] In step S213, in the image sequence, a first image with an expression coefficient greater than the first preset value and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value are selected.

[0108] In this step, the first image with an expression and the second image without an expression are selected from the image sequence to participate in subsequent model training. Specifically, the expression change images in the image sequence can be selected by using the expression coefficient. Generally, the greater the value of the expression coefficient, the greater the expression amplitude of the face, i.e., the expression state; the smaller the value of the expression coefficient, the closer the expression amplitude of the face to the non-expression state. Therefore, in the image sequence, the first image with an expression coefficient greater than the first preset value and the second image with an expression coefficient less than or equal to the first preset value are selected.

[0109] ​​​​It should be noted that when the first image and the second image are acquired, there is no acquisition sequence, that is, in the image sequence, the first image can be selected first, and then the second image of the same object as the first image is selected, or the second image can be selected first, and then the first image of the same object as the second image is selected, or the first image and the second image belonging to the same object can be selected at the same time.

[0110] In the embodiment, the image sequence of expression changes can be quickly acquired through the expression editing model, and the first image and the second image can be quickly and accurately determined by selecting the expression coefficients in the image sequence.

[0111] In an embodiment, before the step S212 of inputting the face image into the expression editing model to acquire the image sequence of expression changes, the method further comprises:

[0112] In step S214, a sixth image is acquired, and the sixth image is training data of the expression editing model.

[0113] In this step, the sixth image is the training data of the expression editing model, and the expression coefficient of the sixth image can be any value, that is, the sixth image can be any given expression face image or non-expression face image.

[0114] In step S215, the sixth image is input into the face expression recognition model to determine the expression coefficient corresponding to the sixth image.

[0115] In this step, the face expression recognition model is a pre-trained data model meeting a preset accuracy, which is used for expression recognition of an image to determine the expression coefficient corresponding to the image. Specifically, the closer the expression coefficient is to 0, the closer the image is to a non-expression image.

[0116] In step S216, the expression coefficient corresponding to the sixth image and random noise are input into a second preset model to obtain a seventh image different from the expression coefficient of the sixth image.

[0117] In this step, the second preset model is a pre-built neural network model, which is used for editing face expressions. It should be noted that the random noise mentioned in this step can be directly randomly generated noise or encoded random noise.

[0118] Specifically, as Figure 4As shown, the second preset model is a pre-built StyleGAN2 model (a kind of neural network model), the sixth image X is input into the facial expression recognition model, the expression coefficient E corresponding to the sixth image is obtained, random noise Z is input into multilayer perceptron (MLP, Multilayer Perceptron), the encoding vector W of random noise is obtained, the encoding vector W of random noise and the expression coefficient E corresponding to the sixth image are input into the StyleGAN2 model, the seventh image X' generated by the StyleGAN2 model is obtained, and the expression coefficients of the sixth image X and the seventh image X' are different.

[0119] In step S217, a second loss value is determined based on the sixth image and the seventh image, and a model parameter of the second preset model is adjusted based on the second loss value to obtain the expression editing model.

[0120] In this step, the sixth image is a real image, and the seventh image is a generated image. The second loss value determined according to the sixth image and the seventh image is used to adjust the model parameter of the second preset model, so as to effectively train the expression editing model with high precision.

[0121] In an embodiment, before the step of determining a second loss value based on the sixth image and the seventh image, adjusting a model parameter of the second preset model based on the second loss value, and obtaining the expression editing model, the method further includes: inputting the seventh image into the facial expression recognition model to obtain an expression coefficient corresponding to the seventh image; determining a first sub-loss value as a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image; determining a second sub-loss value as an average loss corresponding to the sixth image and the seventh image; determining a third sub-loss value as a determination result of the same object corresponding to the sixth image and the seventh image; and determining a second loss value of the sixth image and the seventh image as a calculation result of the first sub-loss value, the second sub-loss value and the third sub-loss value.

[0122] In this embodiment, the second loss value is composed of the first sub-loss value, the second sub-loss value and the third sub-loss value. Specifically, as shown in FIG. 6, the first sub-loss value is determined as a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image, the second sub-loss value is determined as an average loss corresponding to the sixth image and the seventh image, and the third sub-loss value is determined as a determination result of the same object corresponding to the sixth image and the seventh image. Figure 4As shown, the seventh image X' is input into the facial expression recognition model, the facial expression recognition model outputs the expression coefficient E' of the seventh image, the seventh image X' is a generated image, the sixth image X is a real image, the expression coefficient corresponding to the sixth image X and the expression coefficient corresponding to the seventh image X' are substituted into a preset formula to determine the first sub-loss value, that is, the L1 loss, wherein the preset formula can be set according to the actual application scene; the average loss (Adv loss) of the sixth image and the seventh image is calculated, and the average loss is determined as the second sub-loss value; whether the sixth image and the seventh image belong to the same person object is judged, and the third sub-loss value is determined according to the judgment result, which corresponds to Figure 4 the content of the other loss. Further, the first sub-loss value, the second sub-loss value and the third sub-loss value are collectively determined to obtain the second loss value, for example, the sum of the first sub-loss value, the second sub-loss value and the third sub-loss value is determined as the second loss value; of course, the weight coefficients of the first sub-loss value, the second sub-loss value and the third sub-loss value can also be determined, and the weighted average of the first sub-loss value, the second sub-loss value and the third sub-loss value and the respective weight coefficients is determined as the second loss value. Therefore, in the embodiment, the second loss value comprehensively considers the first sub-loss value, the second sub-loss value and the third sub-loss value, which can accurately evaluate the sixth image and the seventh image, and is beneficial to obtain an expression editing model with high precision.

[0123] In step S22, a first target region is cropped from the first image to obtain a third image; the first target region is a local face region containing a pre-set face organ and embodying the identity feature of the object.

[0124] In this step, a local face region corresponding to a face organ is pre-set, which can embody the identity feature of the object, for example, part of the facial feature, which can be used to identify the identity of the object and distinguish it from other objects. A first target region is cropped from the first image to obtain a third image, which can be used to provide the identity feature.

[0125] Specifically, the face region is divided into an upper half face region and a lower half face region, and a face organ is pre-determined, and the half face region where the face organ is located is the local face region, for example, if the face organ is the mouth, the local face region is the lower half face region. Of course, the local face region can also be determined by setting key points, for example, the local face region is pre-determined to be the lower half face region in the face, and a first key point can be selected at the nose and the lower jaw of the face, respectively, and the region composed of the first key points contains the face organ mouth, so that after the first image is obtained, key point detection is performed on the first image, and the lower half face region corresponding to the first key point is determined according to the detection result, and the lower half face region is cropped as the third image.Figure 1 The left image in FIG. 1 is the first image, and the right image is the second image. Figure 1 The step S21 includes: performing key point detection on the first image to determine a first target region corresponding to a first key point, and cropping the first target region to obtain a third image as shown in FIG. 3. Figure 5 The step S21 includes: performing key point detection on the first image to determine a first target region corresponding to a first key point, and cropping the first target region to obtain a third image as shown in FIG. 3.

[0126] In step S23, the second target region in the first image is subjected to occlusion processing to obtain a fourth image; the second target region is a human face organ region corresponding to a human face organ; and the first target region contains the second target region.

[0127] In this step, because a human face expression is usually expressed by a human face organ, for example, when a person is in a very happy state, the person usually laughs with an open mouth, and at this time, the mouth is opened with a large amplitude; when a person is in a calm natural state, the person is usually in a natural state of closing the mouth and opening the eyes. Therefore, a human face organ region corresponding to a human face organ is usually a multi-expression region, has rich expression states, embodies changes in a human face expression, and has a greater impact on an expression coefficient. In the first image, the second target region is subjected to occlusion processing to obtain a fourth image, and the fourth image can effectively remove the expression features of the human face organ in the second target region. The first target region is a local human face region, the second target region is a human face organ region, the area of the first target region is greater than the area of the second target region, and the first target region includes the second target region. For example, the first target region is a lower half face region, and the second target region is a mouth region.

[0128] Specifically, a second key point corresponding to a human face organ region is set in advance, and the second target region is determined in the first image according to the second key point. The human face organ region can be a mouth region, and the second key point can be a mouth key point. For example, in the case where the first target region is a lower half face region and the second target region is a mouth region, the second key point can be set as a mouth key point. The first image is subjected to key point detection to determine the second key point in the first image, the second target region is determined according to the second key point, and the second target region in the first image is subjected to occlusion to obtain a fourth image as shown in FIG. 4, and the fourth image no longer contains features in the second target region. Figure 5

[0129] In an embodiment, the step S23 of performing occlusion processing on the second target region in the first image to obtain a fourth image includes:

[0130] In step S231, in a case where the expression coefficient corresponding to the first image is less than or equal to a second preset value, the pixel value in the second target region in the first image is adjusted to a third preset value for pixel occlusion, which is set in advance, to perform occlusion processing on the second target region to obtain a fourth image; and the second preset value is greater than the first preset value.​

[0131] In this step, the expression coefficient of the first image is greater than the first preset value, that is, the first image is an expression image, but the facial expression amplitude change range is large, so there will be exaggerated expression images in the expression image, such as images with a large mouth. Therefore, the second preset value is used to further divide the expression image. In the case where the expression coefficient corresponding to the first image is less than or equal to the second preset value, it is indicated that the first image can be directly used for occlusion processing, that is, the facial expression of the first image is not exaggerated. A third preset value for pixel occlusion is set, and the pixel value in the second target region in the first image is set to the third preset value, where the third preset value can be 0, so that the second target region in the first image is occluded to obtain a fourth image. In this embodiment, the first image with non-exaggerated expression is screened out by the second preset value, and the second target region in the first image is occluded by the third preset value, effectively removing the expression features corresponding to the second target region in the first image.

[0132] Step S232, in the case where the expression coefficient corresponding to the first image is greater than the second preset value, the first image is animated to obtain a new first image; the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value.

[0133] In this step, if the expression coefficient corresponding to the first image is greater than the second preset value, it is indicated that the first image cannot be directly occluded, that is, the facial expression of the first image is exaggerated. At this time, if the second target region in the first image is directly occluded, a large number of features may be lost. Therefore, the first image is animated to obtain a new first image, and the expression coefficient of the first image is reduced by animation processing, so that the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value. Specifically, CG (computer animation) can be used for animation processing. For example, a first image as shown in FIG. 10a is obtained, the mouth expression of the first image is exaggerated, the expression coefficient corresponding to the first image is determined, and the expression coefficient corresponding to the first image is greater than the second preset value. Then, the first image is processed by CG technology (corresponding to the CGwarp module in FIG. 10b, the CGwarp module is specifically used for computer animation processing and warping processing of the mouth region in the first image), and a new first image as shown in FIG. 10b is obtained. The mouth movement of the new first image is obviously reduced, effectively reducing the occlusion area to retain more features in the fourth image. Figure 6 Figure 6 Figure 6

[0134] ​​​In an embodiment, the animation processing of the first image to obtain a new first image comprises: three-dimensional reconstruction of the first image to determine a three-dimensional face model; adjusting the expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determining a new first image based on the three-dimensional face model with the adjusted expression coefficient.

[0135] Specifically, monocular three-dimensional reconstruction is performed on the first image to obtain a three-dimensional face model corresponding to the first image, the expression coefficient of the three-dimensional face model is adjusted to be less than or equal to the second preset value and greater than the first preset value to obtain an adjusted three-dimensional face model, and an orthographic view of the adjusted three-dimensional face model is obtained to obtain a new first image. By performing three-dimensional reconstruction on the first image, the expression coefficient of the first image is accurately adjusted in three dimensions, and a more realistic animation processing effect is obtained.

[0136] In a possible implementation manner, as Figure 7 described, a target image is obtained, the target image is an image with a smaller given expression, such as an expression coefficient of the target image being less than or equal to the second preset value and greater than the first preset value, a first image is taken as a source image, monocular reconstruction is performed on the first image and the target image to obtain three-dimensional face models corresponding to the first image and the target image respectively, that is, expression coefficients, geometry information, and pose information corresponding to the first image and the target image are obtained, the expression coefficient of the three-dimensional face model of the first image is adjusted to the expression coefficient of the three-dimensional face model of the target image to change the three-dimensional face model of the first image, the three-dimensional face models of the first image and the target image with the changed expression coefficients are respectively reconstructed into meshes, and 3D-Aware Keypoint Alignment is used to align the poses of the two three-dimensional face models after the mesh reconstruction, and then the three-dimensional face model of the first image with the changed expression coefficient is rendered with CG teeth, and the three-dimensional face model after the tooth animation processing is further projected in two dimensions to obtain a new first image, so that CG warp and rendering are used to complete the shape change to obtain the new first image. Figure 7

[0137] In step S233, the pixel value in the second target region in the new first image is adjusted to the third preset value to perform occlusion processing on the second target region to obtain a fourth image. ​

[0138] In this step, after obtaining the new first image, the expression coefficient of the new first image is less than or equal to the second preset value and greater than the first preset value. Therefore, the pixel value of the second target area in the new first image is set to the third preset value to occlude the second target area in the new first image, thus obtaining the fourth image.

[0139] In this embodiment, the expression coefficient corresponding to the first image is determined. The first image with an expression coefficient less than or equal to the second preset value is directly processed. The first image with an expression coefficient greater than the second preset value is first animated to obtain a new first image. Then, the new first image is occluded to obtain a fourth image. This ensures that when the facial expression is exaggerated, the size of the occluded area in the fourth image is reduced, and more facial attribute information is preserved.

[0140] In step S24, the third image and the fourth image are input into the first preset model to obtain a fifth image with a different expression coefficient than the first image.

[0141] In this step, the first preset model is a pre-built neural network model. The first preset model is used to adjust the expression coefficients and generate a fifth image. The expression coefficients of the fifth image are different from those of the first image.

[0142] In one embodiment, the first preset model includes a first neural network, a second neural network, and a third neural network; step S24 inputs the third image and the fourth image into the first preset model to obtain a fifth image with an expression coefficient different from that of the first image, including:

[0143] In step S241, the third image is input into the first neural network for feature extraction to obtain identity feature information in the third image, which is used to distinguish different objects.

[0144] In this step, the first neural network is used to extract features from the third image and determine identity feature information, which is used to distinguish different objects.

[0145] For example, such as Figure 5 The first preset model is the model structure of AEI-Net (a neural network model), wherein the first neural network is... Figure 5 The identity encoder network inputs the third image of the pouting face into the identity encoder network to obtain the z-axis. idThat is, the extracted identity feature information. Specifically, after the lower half face region of the expression image is cropped, the cropped image is resized to 128x128, a third image of 128x128 is input, the first neural network identity encoder is used to upsample the third image, and 512-dimensional identity feature information z is obtained id .

[0146] In step S242, the fourth image is input into the second neural network for feature extraction, to obtain the face attribute information of the object outside the face organ, which is used to reflect the face features in the fourth image.

[0147] In this step, the second neural network is used to extract features from the fourth image to obtain face attribute information, which is used to reflect the face features in the fourth image, including but not limited to head posture, facial expression, skin color, illumination, and background, etc. Because the face organ region corresponding to the face organ in the fourth image is occluded, the obtained face attribute information does not include the face attribute information corresponding to the face organ in the second target region.

[0148] For example, the second neural network is a multi-level attribute encoder (Multi-level Attributes Encoder) as shown in Figure 5 After the fourth image is input into the second neural network, the second neural network outputs multiple different dimensions of face attribute information corresponding to Figure 5

[0149] In step S243, the identity feature information and the face attribute information are input into the third neural network for feature fusion, to obtain a fifth image different from the expression coefficient of the first image.

[0150] In this step, the third neural network is used to fuse the identity feature information and the face attribute information, and output a fifth image different from the expression coefficient of the first image. In this embodiment, the first neural network and the second neural network are used to extract different features respectively, to ensure the accuracy of feature extraction, and the third neural network is used to fuse the identity feature information and the face attribute information, to generate a fifth image with natural and accurate effect.

[0151] For example, the third neural network is an AAD generator (AAD Generator, where AAD is Adaptive Attentional Denormalization) as shown in Figure 5 The AAD generator is used to generate face identity information z​id and face attribute information The AAD residual block (AAD ResBlk) is fused to obtain a fifth image.

[0152] In step S25, a first loss value is determined based on the second image and the fifth image, and a model parameter of the first preset model is adjusted based on the first loss value to obtain an expression adjustment model.

[0153] In this step, the second image is taken as a real image, and the fifth image is taken as a predicted image. A first loss value of the second image and the fifth image is determined, and a model parameter of the first preset model is adjusted according to the first loss value to obtain an expression adjustment model meeting a preset accuracy.

[0154] For example, a preset loss function corresponding to the second image and the fifth image is determined in advance, the first loss value is determined according to the second image, the fifth image and the preset loss function, the model parameter is adjusted according to the first loss value, and the expression adjustment model is trained. The expression adjustment effect of the expression adjustment model is as shown in c of FIG. 8 and c of FIG. 9. Figure 5 Figure 6 c of FIG. 8 and c of FIG. 9 Figure 6 The expression adjustment model corresponding to the expression changeless module in c Figure 5 c and Figure 1 The right image in c is compared, Figure 1 The mouth of the object in the right image in c still presents a pouting effect, Figure 5 The mouth of the object in c presents a natural state without expression, and it can be known that the expression adjustment model provided in this embodiment has a good removal effect.

[0155] ​In the embodiment, the first preset value is used to accurately classify the face expression amplitude, and then the first image and the second image belonging to different classifications are accurately obtained, that is, the expression coefficient of the object in the first image is greater than the first preset value, and the expression coefficient of the object in the second image is less than or equal to the first preset value. A first target region containing a pre-set face organ and embodying the identity feature of the object is determined in the first image, so as to crop the first target region in the first image to obtain a third image, and the third image can effectively embody the identity feature of the object; a second target region corresponding to the face organ is determined in the first image, wherein the first target region includes the second target region, and the second target region is subjected to a shielding process in the first image to obtain a fourth image. The face expression amplitude is usually expressed by the face organ, and the second target region is subjected to the shielding process to effectively remove the expression feature expressed by the face organ of the second target region in the fourth image; further, the third image and the fourth image are input into a first preset model to obtain a fifth image different from the expression coefficient of the first image. Taking the second image with the expression coefficient less than or equal to the first preset value as a learning target, a first loss value corresponding to the second image and the fifth image is determined, and the model parameters of the first preset model are adjusted according to the first loss value to obtain an expression adjustment model. The expression adjustment model provides the identity feature of the object through the third image and effectively removes the expression feature expressed by the face organ of the second target region, reduces the influence of the expression feature expressed by the face organ of the second target region on the fifth image, avoids the possibility that the expression coefficient of the obtained fifth image is still large, and further ensures that the expression adjustment model can effectively reduce the expression coefficient of the object when performing expression adjustment, and obtains a better adjustment effect.

[0156] Figure 8 An expression adjustment method is shown according to an exemplary embodiment, and specifically includes the following steps:

[0157] In step 81, a to-be-processed image is obtained;

[0158] In step 82, in a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, a first target region is cropped in the to-be-processed image to obtain a cropped image, and a second target region is subjected to a shielding process in the to-be-processed image to obtain a to-be-processed image in which the second target region is shielded; the first target region is a local face region of a pre-set face organ and embodies an identity feature of an object, and the second target region is a face organ region corresponding to the face organ; the first target region contains the second target region;

[0159] In step 83, the cropped image and the to-be-processed image in which the second target region is shielded are input into an expression adjustment model to obtain a target image; an expression coefficient of the target image is less than or equal to the preset value;

[0160] The expression adjustment model is obtained according to the training method of the expression adjustment model.

[0161] In this embodiment, a to-be-processed image is obtained, the to-be-processed image being an arbitrary given face image, which can be an expression image or an expressionless image. An expression coefficient corresponding to the to-be-processed image is determined. A preset value is set in advance to filter the to-be-processed image. In a case where the expression coefficient corresponding to the to-be-processed image is greater than the preset value, the expression of the to-be-processed image is relatively large, and thus the to-be-processed image is subjected to cropping processing and occlusion processing. Specifically, a local face region corresponding to a first target region and a face organ region corresponding to a second target region are set in advance to determine the first target region and the second target region in the to-be-processed image. The first target region is cropped in the to-be-processed image to obtain a cropped image. The second target region in the to-be-processed image is subjected to occlusion processing to obtain a to-be-processed image in which the second target region is occluded. The cropped image and the to-be-processed image in which the second target region is occluded are input into the trained expression adjustment model. The expression of the to-be-processed image is effectively removed by the expression adjustment model to obtain a target image in which the expression coefficient is less than or equal to the preset value. The expression adjustment effect of the target image is relatively good.

[0162] In a possible implementation, a second preset value is determined. If the expression coefficient of the to-be-processed image is greater than the preset value and less than or equal to the second preset value, the pixel values in the second target region in the to-be-processed image are all set to a third set value to occlude the second target region, to obtain a to-be-processed image in which the second target region is occluded. The third set value can be 0. If the expression coefficient of the to-be-processed image is greater than the second preset value, the to-be-processed image is subjected to animation processing to obtain a new to-be-processed image. The expression coefficient corresponding to the new to-be-processed image is greater than the preset value and less than or equal to the second preset value. The pixel values in the second target region in the new to-be-processed image are all set to the third set value to occlude the second target region, to obtain an occluded image.

[0163] Further, the animation processing on the to-be-processed image to obtain a new to-be-processed image includes: performing three-dimensional reconstruction on the to-be-processed image to determine a three-dimensional face model; adjusting the expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the preset value; and determining the new to-be-processed image based on the three-dimensional face model with the adjusted expression coefficient.

[0164] Figure 9 An expression adjustment model training apparatus according to an example embodiment is shown. The apparatus includes an image acquisition module 91, a cropping processing module 92, an occlusion processing module 93, an image generation module 94, and a parameter adjustment module 95.

[0165] The image acquisition module 91 is configured to acquire a first image and a second image of a same object; an expression coefficient of the object in the first image is greater than a first preset value, and an expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to represent a facial expression amplitude in an image; and the first preset value is a threshold value preset for binary classification of the facial expression amplitude.

[0166] The cropping processing module 92 is configured to crop a first target region in the first image to obtain a third image; the first target region includes a local facial region of a preset facial organ and embodies an identity feature of the object.

[0167] The occlusion processing module 93 is configured to perform occlusion processing on a second target region in the first image to obtain a fourth image; the second target region is a facial organ region corresponding to the facial organ; and the first target region includes the second target region.

[0168] The image generation module 94 is configured to input the third image and the fourth image into a first preset model to obtain a fifth image with a different expression coefficient from the first image.

[0169] The parameter adjustment module 95 is configured to determine a first loss value based on the second image and the fifth image, and adjust model parameters of the first preset model based on the first loss value to obtain an expression adjustment model.

[0170] In an example embodiment, the occlusion processing module includes:

[0171] A first occlusion processing unit is configured to, in a case where the expression coefficient corresponding to the first image is less than or equal to a second preset value, adjust a pixel value in a second target region in the first image to a third preset value preset for pixel occlusion to perform occlusion processing on the second target region to obtain a fourth image; and the second preset value is greater than the first preset value.

[0172] In an example embodiment, the occlusion processing module further includes:

[0173] A second occlusion processing unit is configured to, in a case where the expression coefficient corresponding to the first image is greater than the second preset value, perform animation processing on the first image to obtain a new first image; the new first image corresponds to an expression coefficient less than or equal to the second preset value and greater than the first preset value; and adjust a pixel value in a second target region in the new first image to the third preset value to perform occlusion processing on the second target region to obtain a fourth image.

[0174] In an example embodiment, the second occlusion processing unit comprises:

[0175] The reconstruction processing sub-unit is configured to perform three-dimensional reconstruction on the first image to determine a three-dimensional face model;

[0176] The image determination sub-unit is configured to adjust an expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determine a new first image based on the three-dimensional face model after adjustment of the expression coefficient.

[0177] In an example embodiment, the image generation module comprises:

[0178] The first feature extraction unit is configured to input the third image into the first neural network for feature extraction to obtain identity feature information in the third image, the identity feature information being used to distinguish different objects;

[0179] The second feature extraction unit is configured to input the fourth image into the second neural network for feature extraction to obtain face attribute information of the object outside the face organ, the face attribute information being used to reflect face features in the fourth image;

[0180] The feature fusion processing unit is configured to input the identity feature information and the face attribute information into the third neural network for feature fusion to obtain a fifth image different from the expression coefficient of the first image.

[0181] In an example embodiment, the image acquisition module comprises:

[0182] The image acquisition unit is configured to acquire a face image;

[0183] The image editing unit is configured to input the face image into an expression editing model to acquire an image sequence of expression changes;

[0184] The image selection unit is configured to select, from the image sequence, a first image with an expression coefficient greater than the first preset value, and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value.

[0185] In an example embodiment, the device further comprises an editing model determination module, the editing model determination module comprising:

[0186] The training data acquisition unit is configured to acquire a sixth image, the sixth image being training data of the expression editing model;

[0187] The expression coefficient determination unit is configured to input the sixth image into a facial expression recognition model to determine an expression coefficient corresponding to the sixth image.

[0188] The expression coefficient adjustment unit is configured to input the expression coefficient corresponding to the sixth image and random noise into a second preset model to obtain a seventh image different from the expression coefficient of the sixth image.

[0189] The editing model determination unit is configured to determine a second loss value based on the sixth image and the seventh image, and adjust a model parameter of the second preset model based on the second loss value to obtain the expression editing model.

[0190] In an example embodiment, the editing model determination module further includes a loss value determination unit. The loss value determination unit is configured to input the seventh image into the facial expression recognition model to obtain an expression coefficient corresponding to the seventh image, determine a first sub-loss value based on a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image, determine a second sub-loss value based on an average loss corresponding to the sixth image and the seventh image, determine a third sub-loss value based on a determination result of the same object corresponding to the sixth image and the seventh image, and determine a second loss value of the sixth image and the seventh image based on a calculation result of the first sub-loss value, the second sub-loss value, and the third sub-loss value.

[0191] Figure 10 An expression adjustment device is shown according to an example embodiment. The device includes an image acquisition module 101, an image processing module 102, and a model processing module 103.

[0192] The image acquisition module 101 is configured to acquire a to-be-processed image.

[0193] The image processing module 102 is configured to, in a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, crop a first target region from the to-be-processed image to obtain a cropped image, and perform occlusion processing on a second target region in the to-be-processed image to obtain a to-be-processed image in which the second target region is occluded. The first target region is a local face region of a pre-set face organ and embodies an identity feature of an object, and the second target region is a face organ region corresponding to the face organ. The first target region contains the second target region.

[0194] The model processing module 103 is configured to input the cropped image and the to-be-processed image in which the second target region is occluded into an expression adjustment model to obtain a target image. An expression coefficient of the target image is less than or equal to the preset value.

[0195] The expression adjustment model is obtained according to the training method of the expression adjustment model.

[0196] As to the apparatus in the above embodiments, the specific manners in which the respective modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0197] Figure 11 is a block diagram of an electronic device according to an example embodiment, which can be a terminal, a server or similar computing device. Take the electronic device 1100 as a server for example. Referring to Figure 11 , the electronic device 1100 includes a processing component 1122, which further includes one or more processors, and a memory resource represented by a memory 1132, for storing instructions, such as an application program, executable by the processing component 1122. The application program stored in the memory 1132 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1122 is configured to execute the instructions to perform the training method of the expression adjustment model or the expression adjustment method.

[0198] The electronic device 1100 can further include a power supply component 1126 configured to perform power management of the electronic device 1100, a wired or wireless network interface 1150 configured to connect the electronic device 1100 to a network, and an input / output (I / O) interface 1158. The electronic device 1100 can operate based on an operating system stored in the memory 1132, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like.

[0199] In an example embodiment, a computer readable storage medium including instructions, such as the memory 1132 including instructions, is also provided, which can be executed by the processing component 1122 of the electronic device 1100 to complete the implementation method of the training method of the expression adjustment model or the expression adjustment method. Optionally, the computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk and an optical data storage device, etc.

[0200] In an example embodiment, a computer program product including a computer program or computer instructions is also provided, which, when executed by a processor, implements the training method of the expression adjustment model or the expression adjustment method.

[0201] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0202] It should be understood that the application is not limited to the precise construction hereinafter described and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A training method for an expression adjustment model, characterized in that, The method comprises: obtaining a first image and a second image corresponding to the same object; an expression coefficient of the object in the first image is greater than a first preset value, and an expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to represent the facial expression amplitude in the image; the first preset value is a threshold value preset for binary classification of the facial expression amplitude; a first target region is cropped in the first image to obtain a third image; the first target region is a local face region containing a preset face organ and embodying the identity feature of the object; a second target region is subjected to occlusion processing in the first image to obtain a fourth image; the second target region is a face organ region corresponding to the face organ; the first target region contains the second target region; the third image and the fourth image are input into a first preset model to obtain a fifth image different from the expression coefficient of the first image; a first loss value is determined based on the second image and the fifth image, and a model parameter of the first preset model is adjusted based on the first loss value to obtain an expression adjustment model; the first preset model comprises a first neural network, a second neural network and a third neural network; the third image and the fourth image are input into the first preset model to obtain the fifth image different from the expression coefficient of the first image, which comprises: the third image is input into the first neural network for feature extraction to obtain identity feature information in the third image, which is used to distinguish different objects; the fourth image is input into the second neural network for feature extraction to obtain face attribute information of the object outside the face organ, which is used to represent the face feature in the fourth image; the identity feature information and the face attribute information are input into the third neural network for feature fusion to obtain the fifth image different from the expression coefficient of the first image.

2. The method of claim 1, wherein, the second target region is subjected to occlusion processing in the first image to obtain the fourth image, which comprises: in a case where the expression coefficient corresponding to the first image is less than or equal to a second preset value, a pixel value in the second target region in the first image is adjusted to a third preset value preset for pixel occlusion to perform occlusion processing on the second target region to obtain the fourth image; the second preset value is greater than the first preset value.

3. The method of claim 2, wherein, The method further comprises: in a case where the expression coefficient corresponding to the first image is greater than the second preset value, the first image is subjected to animation processing to obtain a new first image; the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value; a pixel value in the second target region in the new first image is adjusted to the third preset value to perform occlusion processing on the second target region to obtain the fourth image.

4. The method of claim 3, wherein, the first image is subjected to animation processing to obtain the new first image, which comprises: three-dimensional reconstruction is performed on the first image to determine a three-dimensional face model; Adjust the expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determine a new first image based on the three-dimensional face model after the expression coefficient is adjusted.

5. The method of claim 1, wherein, The method comprises: Obtaining a face image; Inputting the face image into an expression editing model to obtain an image sequence of expression changes; In the image sequence, a first image with an expression coefficient greater than the first preset value is selected, and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value is selected.

6. The method of claim 5, wherein, Before the step of inputting the face image into the expression editing model to obtain an image sequence of expression changes, the method further comprises: Obtaining a sixth image, which is training data of the expression editing model; Inputting the sixth image into a face expression recognition model to determine an expression coefficient corresponding to the sixth image; Inputting the expression coefficient corresponding to the sixth image and random noise into a second preset model to obtain a seventh image different from the expression coefficient of the sixth image; Determining a second loss value based on the sixth image and the seventh image, adjusting model parameters of the second preset model based on the second loss value, and obtaining the expression editing model.

7. The method of claim 6, wherein, Before the step of determining a second loss value based on the sixth image and the seventh image, adjusting model parameters of the second preset model based on the second loss value, and obtaining the expression editing model, the method further comprises: Inputting the seventh image into the face expression recognition model to obtain an expression coefficient corresponding to the seventh image; Determining a first sub-loss value based on a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image; Determining a second sub-loss value based on an average loss corresponding to the sixth image and the seventh image; Determining a third sub-loss value based on a determination result of the sixth image and the seventh image belonging to the same object; Determining a calculation result of the first sub-loss value, the second sub-loss value, and the third sub-loss value as the second loss value of the sixth image and the seventh image.

8. An expression adjustment method characterized by comprising: The method comprises: Obtaining a to-be-processed image; In a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, a first target region is cropped from the to-be-processed image to obtain a cropped image, and a second target region is subjected to occlusion processing in the to-be-processed image to obtain a to-be-processed image in which the second target region is occluded; the first target region is a local face region of a pre-set face organ and embodies an identity feature of an object, and the second target region is a face organ region corresponding to the face organ; the first target region contains the second target region; Inputting the cropped image and the to-be-processed image in which the second target region is occluded into an expression adjustment model to obtain a target image; an expression coefficient of the target image is less than or equal to the preset value; The expression adjustment model is obtained according to the training method of the expression adjustment model in any one of claims 1-7. 9.A device for training an expression adjustment model, comprising: The method comprises: The image acquisition module is configured to acquire a first image and a second image corresponding to the same object; The expression coefficient of the object in the first image is greater than a first preset value, and the expression coefficient of the object in the second image is less than or equal to the first preset value; the expression coefficient is used to represent the facial expression amplitude in the image; The first preset value is a threshold value preset for binary classification of the facial expression amplitude; The cropping processing module is configured to crop a first target region in the first image to obtain a third image; the first target region includes a local face region of a preset face organ and embodies the identity feature of the object; The occlusion processing module is configured to perform occlusion processing on a second target region in the first image to obtain a fourth image; the second target region is a face organ region corresponding to the face organ; and the first target region includes the second target region. The image generation module is configured to input the third image and the fourth image into a first preset model to obtain a fifth image different from the expression coefficient of the first image; the first preset model includes a first neural network, a second neural network and a third neural network; The parameter adjustment module is configured to determine a first loss value based on the second image and the fifth image, and adjust model parameters of the first preset model based on the first loss value to obtain an expression adjustment model. The image generation module includes: The first feature extraction unit is configured to input the third image into the first neural network for feature extraction to obtain identity feature information in the third image, and the identity feature information is used to distinguish different objects; The second feature extraction unit is configured to input the fourth image into the second neural network for feature extraction to obtain face attribute information of the object outside the face organ, and the face attribute information is used to embody the face feature in the fourth image; The feature fusion processing unit is configured to input the identity feature information and the face attribute information into the third neural network for feature fusion to obtain a fifth image different from the expression coefficient of the first image.

10. The apparatus of claim 9, wherein, The occlusion processing module includes: The first occlusion processing unit is configured to adjust pixel values in the second target region in the first image to a third preset value preset for pixel occlusion to perform occlusion processing on the second target region and obtain a fourth image when the expression coefficient corresponding to the first image is less than or equal to a second preset value; the second preset value is greater than the first preset value.

11. The apparatus of claim 10, wherein, The occlusion processing module further includes: The second occlusion processing unit is configured to perform animation processing on the first image to obtain a new first image when the expression coefficient corresponding to the first image is greater than the second preset value; the expression coefficient corresponding to the new first image is less than or equal to the second preset value and greater than the first preset value; and the pixel values in the second target region in the new first image are adjusted to the third preset value to perform occlusion processing on the second target region to obtain a fourth image.

12. The apparatus of claim 11, wherein, The second occlusion processing unit comprises: The reconstruction processing subunit is configured to perform three-dimensional reconstruction on the first image to determine a three-dimensional face model; The image determination subunit is configured to adjust an expression coefficient corresponding to the three-dimensional face model to be less than or equal to the second preset value and greater than the first preset value, and determine a new first image based on the three-dimensional face model after adjustment of the expression coefficient.

13. The apparatus of claim 9, wherein, The image acquisition module comprises: The image acquisition unit is configured to acquire a face image; The image editing unit is configured to input the face image into an expression editing model to acquire an image sequence of expression changes; The image selection unit is configured to select, from the image sequence, a first image with an expression coefficient greater than the first preset value, and a second image belonging to the same object as the first image and having an expression coefficient less than or equal to the first preset value.

14. The apparatus of claim 13, wherein, The device further comprises an editing model determination module, and the editing model determination module comprises: The training data acquisition unit is configured to acquire a sixth image, which is training data of the expression editing model; The expression coefficient determination unit is configured to input the sixth image into a face expression recognition model to determine an expression coefficient corresponding to the sixth image; The expression coefficient adjustment unit is configured to input the expression coefficient corresponding to the sixth image and random noise into a second preset model to obtain a seventh image different from the expression coefficient of the sixth image; The editing model determination unit is configured to determine a second loss value based on the sixth image and the seventh image, adjust model parameters of the second preset model based on the second loss value, and obtain the expression editing model.

15. The apparatus of claim 14, wherein, The editing model determination module further comprises a loss value determination unit, and the loss value determination unit is configured to input the seventh image into the face expression recognition model to obtain an expression coefficient corresponding to the seventh image; determine a first sub-loss value based on a calculation result of the expression coefficient corresponding to the sixth image and the expression coefficient corresponding to the seventh image; determine a second sub-loss value based on an average loss corresponding to the sixth image and the seventh image; determine a third sub-loss value based on a determination result of the same object corresponding to the sixth image and the seventh image; and determine a second loss value of the sixth image and the seventh image based on a calculation result of the first sub-loss value, the second sub-loss value, and the third sub-loss value.

16. An expression adjustment device, characterized by The image acquisition module is configured to acquire a to-be-processed image; The image processing module is configured to, in a case where an expression coefficient corresponding to the to-be-processed image is greater than a preset value, crop a first target region from the to-be-processed image to obtain a cropped image, and perform occlusion processing on a second target region in the to-be-processed image to obtain a to-be-processed image in which the second target region is occluded; the first target region is a local face region of a pre-set face organ and embodies an identity feature of an object, the second target region is a face organ region corresponding to the face organ, and the first target region contains the second target region; ​ The model processing module is configured to input the cropped image and the image to be processed in which the second target region is occluded into an expression adjustment model to obtain a target image; an expression coefficient of the target image is less than or equal to the preset value. The expression adjustment model is obtained according to the training method of the expression adjustment model in any one of claims 1-7.

17. An electronic device, comprising: The expression adjustment model comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training method of the expression adjustment model in any one of claims 1-7, or the expression adjustment method in claim 8.

18. A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enabling the electronic device to perform the training method of the expression adjustment model in any one of claims 1-7, or the expression adjustment method in claim 8.

Images