Face image expression correction method and device, electronic equipment and storage medium
By using an expression classification and correction model, combined with pixel, convolution, and feature-level correction techniques, the problem of reduced face recognition accuracy caused by expression deformation was solved, achieving efficient correction and improved recognition accuracy of expressionless face images.
Patent Information
- Application Number
- CN202310149418.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-16
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-02-16
AI Technical Summary
Existing technologies suffer from reduced accuracy in facial recognition when processing facial images with facial expressions, especially when the expression distortion is significant, resulting in a large difference between the facial image and the real facial image.
The facial expression category is determined by an expression classification model, and the facial image is corrected into an expressionless image by an expression correction model, including pixel correction, convolution correction and feature-level correction. Combined with adaptive enhancement technology, the final expressionless facial image is obtained.
It effectively reduces the difference between the corrected expressionless face image and the real face image, thus improving the accuracy of face recognition.
Smart Images
Figure CN116434293B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of face recognition, in particular to a face image expression correction method and device, an electronic device and a storage medium. BACKGROUND
[0002] Face image recognition is applied to many scenarios, bringing great convenience to people's life. However, the expression in the face image causes the face to deform, which reduces the accuracy of face image recognition and affects the performance of the face recognition system. Therefore, how to correct the expression in the face image to reduce the deformation caused by the expression in the face image has been a problem to be solved in the industry.
[0003] The method commonly used in the industry is to perform face alignment through local key points of the face image, and to perform face recognition using the aligned face image. When the expression causes small deformation of the face, this alignment method can play a certain role. However, when the expression causes large deformation of the face, the aligned face image will be distorted, and the aligned face image will be significantly different from the real face image. SUMMARY
[0004] Therefore, the purpose of the present application is to provide a face image expression correction method and device, an electronic device and a storage medium, which can correct the face image with expression to a face image without expression, and reduce the difference between the corrected face image without expression and the real face image.
[0005] In a first aspect, the embodiments of the present application provide a face image expression correction method, which includes:
[0006] obtaining a face image and a face expression feature corresponding to the face image;
[0007] inputting the face expression feature corresponding to the face image into an expression classification model to obtain an expression category corresponding to the face image;
[0008] inputting the face image into an expression correction model corresponding to the expression category to obtain a face image without expression after expression correction;
[0009] The expression classification model is obtained by training face expression feature samples and corresponding expression categories, and the expression correction model is obtained by training face image samples and corresponding face images without expression.
[0010] In a possible implementation, inputting the face image into the expression correction model corresponding to the expression category to obtain the face image without expression after expression correction includes:
[0011] The face image features are corrected pixel by pixel by using the face image features and the pixel correction kernel of the expression correction model to obtain the first corrected face image features.
[0012] The first corrected face image features are convolved and corrected based on the convolutional correction kernel of the expression correction model to obtain the second corrected face image features.
[0013] Based on the second corrected face image features and the residual kernel of the expression correction model, feature-level correction is performed on the second corrected face image features to obtain an expressionless face image after expression correction.
[0014] In one possible implementation, the facial image features are pixel-corrected based on the facial image features of the face image and the pixel correction kernel of the expression correction model to obtain the first corrected facial image features, including:
[0015] For each pixel block in the facial image features, the pixel correction kernel is subtracted from the pixel block to obtain the first corrected facial image features.
[0016] In one possible implementation, the first corrected face image features are convolutionally corrected based on the first corrected face image features and the convolutional correction kernel of the expression correction model to obtain the second corrected face image features, including:
[0017] After inputting the first corrected face image features into the convolutional layer and performing convolution, the first corrected face image features after convolution are obtained.
[0018] Subtract the convolution correction kernel from the first corrected face image features after convolution to obtain the second corrected face image features.
[0019] In one possible implementation, the second modified face image features are modified at the feature level based on the second modified face image features and the residual kernel of the expression correction model to obtain an expressionless face image after expression correction, including:
[0020] The following formula is used to obtain the expressionless face image after expression correction;
[0021] F = (1-W) r E;
[0022] Where F is the expressionless face image after expression correction, W r is the residual kernel, and E is the second corrected face image feature.
[0023] In one possible implementation, the method further includes:
[0024] Adaptive enhancement is performed on the expression-corrected expressionless face image to obtain the final expression-corrected expressionless face image.
[0025] In a possible implementation, the expression-corrected expressionless face image is adaptively enhanced to obtain a final expression-corrected expressionless face image, including:
[0026] The expression-corrected expressionless face image is adaptively enhanced by the following formula to obtain the final expression-corrected expressionless face image.
[0027]
[0028] Wherein, K is the final expression-corrected expressionless face image, F is the expressionless face image before adaptive enhancement, f is an element in F, is the Hadamard vector product.
[0029] In a second aspect, the embodiments of the present application further provide a face image expression correction device, the face image expression correction device comprising:
[0030] An acquisition module is configured to acquire a face image and a face expression feature corresponding to the face image.
[0031] An input module is configured to input the face expression feature corresponding to the face image into an expression classification model to obtain an expression category corresponding to the face image.
[0032] The input module is further configured to input the face image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image.
[0033] The expression classification model is obtained by training face expression feature samples and corresponding expression categories, and the expression correction model is obtained by training face image samples and corresponding expressionless face images.
[0034] In a possible implementation, the input module is specifically configured to perform pixel correction on the face image feature according to the face image feature of the face image, a pixel correction kernel of the expression correction model, to obtain a first corrected face image feature; perform convolution correction on the first corrected face image feature according to the first corrected face image feature and a convolution correction kernel of the expression correction model, to obtain a second corrected face image feature; and perform feature-level correction on the second corrected face image feature according to the second corrected face image feature and a residual kernel of the expression correction model, to obtain the expression-corrected expressionless face image.
[0035] In a possible implementation, the input module is further configured to, for each pixel block in the face image feature, subtract the pixel block from the pixel correction kernel to obtain the first corrected face image feature.
[0036] In a possible implementation, the input module is further configured to:
[0037] The first modified face image feature is input into a convolution layer for convolution, and a first modified face image feature after convolution is obtained.
[0038] The first modified face image feature after convolution is subtracted by a convolution modification kernel, and a second modified face image feature is obtained.
[0039] In a possible implementation, the input module is further configured to:
[0040] The expression-modified expressionless face image is obtained through the following formula:
[0041] F = (1 - W r )E;
[0042] wherein F is the expression-modified expressionless face image, W r is a residual kernel, and E is the second modified face image feature.
[0043] In a possible implementation, the apparatus further includes an enhancement module.
[0044] The enhancement module is configured to perform adaptive enhancement on the expression-modified expressionless face image to obtain a final expression-modified expressionless face image.
[0045] In a possible implementation, the enhancement module is specifically configured to perform adaptive enhancement on the expression-modified expressionless face image through the following formula to obtain the final expression-modified expressionless face image: wherein K is the final expression-modified expressionless face image, F is the expressionless face image before adaptive enhancement, f is an element in F, is a Hadamard vector product.
[0046] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor, a storage medium, and a bus. The storage medium stores machine readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium through the bus. The processor executes the machine readable instructions to perform the steps of the face image expression modification method in any one of the first aspect.
[0047] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the face image expression modification method in any one of the first aspect are performed.
[0048] The embodiment of the present application provides a facial expression correction method, device, electronic equipment and storage medium of a facial image, and the facial expression correction method comprises the following steps: acquiring a facial image and a facial expression feature corresponding to the facial image; inputting the facial expression feature corresponding to the facial image into an expression classification model to obtain an expression category corresponding to the facial image; inputting the facial image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless facial image; wherein the expression classification model is obtained by training facial expression feature samples and corresponding expression categories; and the expression correction model is obtained by training facial image samples and corresponding expressionless facial images. The expression classification model is used to determine the expression category of the facial image, and then the expressionless facial image of the facial image is determined according to the expression correction model corresponding to the expression category, so that the facial image with expression can be corrected into the expressionless facial image, and the difference between the expressionless facial image after correction and the real facial image is reduced. BRIEF DESCRIPTION OF DRAWINGS
[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.
[0050] Figure 1 A flow chart of a facial expression correction method provided by the embodiment of the present application is shown;
[0051] Figure 2 A flow chart of another facial expression correction method provided by the embodiment of the present application is shown;
[0052] Figure 3 A structural schematic diagram of a facial expression correction device provided by the embodiment of the present application is shown;
[0053] Figure 4 A structural schematic diagram of an electronic equipment provided by the embodiment of the present application is shown. DETAILED DESCRIPTION
[0054] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions of the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportion. The flowchart used in the present application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or one or more operations can be removed from the flowchart under the guidance of the content of the present application.
[0055] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0056] In order to enable those skilled in the art to use the content of the present application, the following implementation is given in combination with a specific application scenario "face recognition technology field". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of the present application. Although the present application is mainly described around "face recognition technology field", it should be understood that this is only an exemplary embodiment.
[0057] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0058] The following will be described in detail the face image expression correction method provided by the embodiments of the present application.
[0059] Referring to Figure 1 Fig. 1 is a flowchart of a face image expression correction method provided by the embodiments of the present application, and the following will be described for the exemplary steps of the embodiments of the present application.
[0060] S101, acquiring a face image and a face expression feature corresponding to the face image.
[0061] In this embodiment, the face in the face image is a face with an expression, that is, the face image contains a facial expression. Facial expressions can include happiness, crying, sadness, etc. Facial expression features refer to the feature matrix of facial expressions in the face image.
[0062] In addition, the initial facial expression features of the face image are extracted using the following formula of the expression feature extractor.
[0063]
[0064] Where A2 is the initial facial expression feature, A1 is the facial image feature of the face image, R(A1) is the facial expression feature corresponding to facial image feature A1, and W Q W is a query matrix for facial image features. K W is the key matrix for facial image features. V Let W be the value matrix of the face image features, and m be the dimension of the face image features. Q W K W V These are the preset hyperparameters.
[0065] Here, the facial expression feature extractor consists of a transformer with a multi-layer self-attention mechanism. Q W K W V It is learned by a neural network.
[0066] Then, since facial expression features are sub-features of facial features, their dimensionality should be lower than that of the facial image features. However, the current initial facial expression features have the same dimensionality as the facial image features, resulting in redundant components in the extracted initial facial expression features. Therefore, it is necessary to reduce the dimensionality of the initial facial expression features through expression space projection.
[0067] Here, the dimensionality of the initial facial expression features is reduced using the following formula to obtain the final facial expression features.
[0068] B = aA2 + b; where B is the final facial expression feature, A2 is the initial facial expression feature, a is a preset coefficient with dimension n1*n2, b is a preset parameter with dimension n1, n1 is the dimension of the final facial expression feature, and n2 is the dimension of the initial facial expression feature. n1 is less than n2.
[0069] S102. Input the facial expression features corresponding to the face image into the expression classification model to obtain the expression category corresponding to the face image.
[0070] In the embodiment of the present application, when the face image is obtained, the expression category in the face image cannot be directly known as a person, and the expression category refers to the face expression in the face image being happy, sad, or the like.
[0071] Here, the expression classification model is obtained by training the face expression feature samples and the corresponding expression categories. The specific steps are as follows:
[0072] I. Multiple face image samples and face expression feature samples corresponding to the face image samples; all face expression feature samples are clustered according to the Euclidean distance to obtain the expression categories corresponding to each cluster;
[0073] II. For each cluster, the face expression feature samples in the cluster are combined in pairs to obtain a face expression feature sample group; for each face expression feature sample group, the similarity of two face expression feature samples in the face expression feature sample group is determined; if the similarity is less than a first preset value, the two face expression feature samples are removed from the cluster to obtain an intermediate cluster.
[0074] III. The intermediate clusters are combined in pairs to obtain an intermediate cluster group; for each intermediate cluster group, the similarity between the center points of two intermediate clusters in the intermediate cluster group is determined; if the similarity between the center points of the two intermediate clusters is less than a second preset value, the intermediate cluster containing fewer face expression feature samples is removed from the two intermediate clusters to obtain a target cluster.
[0075] IV. The face expression feature samples in the target cluster are used as training samples, and the expression categories corresponding to the face expression features in the target cluster are used as labels to train the expression classification model.
[0076] Here, the second and third steps increase the distance within the cluster and reduce the distance between the clusters, which greatly improves the accuracy and reliability of the expression classification model.
[0077] S103. Input the face image into the expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image.
[0078] In the embodiment of the present application, the expression correction model is used to correct the expression in the face image, and corrects the expression face image into an expressionless face image. Different expression correction models correct different types of expressions.
[0079] The expression correction model is obtained by training the face image samples and the corresponding expressionless face images.
[0080] Specifically, the pixel correction kernel is used to correct the pixel blocks of the face image features according to the face image features of the face image and the expression correction model, to obtain first corrected face image features.
[0081] Here, for each pixel block in the face image features, the pixel block is subtracted from the pixel correction kernel to obtain the first corrected face image features.
[0082] In the embodiments of the present application, the pixel correction kernel is a preset 3*3 matrix, and the pixel block is a 3*3 matrix.
[0083] For example, the face image features are C, wherein C1, C2, C3, C4 are pixel blocks. wherein d1, d2, d3, d4, d5, d6, d7, d8, d9 are values of corresponding positions in the face image features. The pixel correction kernel is wherein h1, h2, h3, h4, h5, h6, h7, h8, h9 are preset values of the pixel correction kernel. The pixel block C1 is subtracted from the pixel correction kernel H, C2, C3, C4 are the same.
[0084] Specifically, the first corrected face image features are convoluted according to the convolution correction kernel of the expression correction model, to obtain second corrected face image features.
[0085] Here, after all the first corrected face image features are input into the convolution layer for convolution, the first corrected face image features after convolution are obtained; the second corrected face image features are obtained by subtracting the convolution correction kernel from the first corrected face image features after convolution.
[0086] In the embodiments of the present application, the second corrected face image features are 3*3 matrices. The convolution correction kernel is a preset 3*3 matrix, and the specific calculation process of subtracting the convolution correction kernel from the first corrected face image features after convolution is referred to the calculation process of subtracting the pixel correction kernel H from the pixel block C1.
[0087] Specifically, the second corrected face image features are corrected at the feature level according to the residual kernel of the expression correction model, to obtain an expression-corrected expressionless face image.
[0088] Here, the expression-corrected expressionless face image is obtained by the following formula.
[0089] F=(1-W r )E;
[0090] wherein F is the expression-corrected expressionless face image, W r is the residual kernel, and E is the second corrected face image features.
[0091] The embodiment of the present application provides a facial expression correction method of a facial image, the facial expression correction method of the facial image comprising: acquiring a facial image and a facial expression feature corresponding to the facial image; inputting the facial expression feature corresponding to the facial image into an expression classification model to obtain an expression category corresponding to the facial image; inputting the facial image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless facial image; wherein the expression classification model is obtained by training a facial expression feature sample and a corresponding expression category; and the expression correction model is obtained by training a facial image sample and a corresponding expressionless facial image. The embodiment of the present application determines the expression category of the facial image through the expression classification model, and then determines the expressionless facial image of the facial image according to the expression correction model corresponding to the expression category, so that the facial image with an expression can be corrected into an expressionless facial image, and the difference between the expressionless facial image after correction and the real facial image is reduced.
[0092] Referring to Figure 2 Fig. 2 shows a flowchart of another facial expression correction method of a facial image provided by the embodiment of the present application, and steps S201 to S203 are referred to steps S101 to S103, which will not be repeated here. The exemplary steps of the embodiment of the present application are described as follows.
[0093] S201, acquiring a facial image and a facial expression feature corresponding to the facial image.
[0094] S202, inputting the facial expression feature corresponding to the facial image into an expression classification model to obtain an expression category corresponding to the facial image.
[0095] S203, inputting the facial image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless facial image.
[0096] S204, adaptively enhancing the expression-corrected expressionless facial image to obtain a final expression-corrected expressionless facial image.
[0097] Specifically, the expression-corrected expressionless facial image is adaptively enhanced by the following formula to obtain the final expression-corrected expressionless facial image.
[0098]
[0099] Wherein, K is the final expression-corrected expressionless facial image, F is the expressionless facial image before adaptive enhancement, f is an element in F, is the Hadamard vector product.
[0100] Here, since the image features of the face image are damaged while the expression features in the face image are corrected, self-enhancement is needed to compensate for the loss caused thereby. The self-enhancement is self-adaptive enhancement according to the feature value weight.
[0101] The embodiment of the present application provides an expression correction method for a face image, which comprises the following steps: inputting face expression features corresponding to a face image into an expression classification model to obtain an expression category corresponding to the face image; inputting the face image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image; and performing self-adaptive enhancement on the expression-corrected expressionless face image to obtain a final expression-corrected expressionless face image. The expression-corrected expressionless face image can be obtained from the face image with expression, and the difference between the expression-corrected expressionless face image and the real face image is reduced.
[0102] Referring to Figure 3 As shown in the figure, the expression correction device for a face image provided by the embodiment of the present application comprises:
[0103] The acquisition module 301 is configured to acquire a face image and face expression features corresponding to the face image.
[0104] The input module 302 is configured to input the face expression features corresponding to the face image into an expression classification model to obtain an expression category corresponding to the face image.
[0105] The input module 302 is further configured to input the face image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image.
[0106] The expression classification model is obtained by training face expression feature samples and corresponding expression categories, and the expression correction model is obtained by training face image samples and corresponding expressionless face images.
[0107] In a possible implementation, the input module 302 is specifically configured to perform pixel correction on the face image features according to the face image features of the face image, pixel correction kernels of the expression correction model, to obtain first corrected face image features; perform convolution correction on the first corrected face image features according to the first corrected face image features and convolution correction kernels of the expression correction model, to obtain second corrected face image features; and perform feature-level correction on the second corrected face image features according to the second corrected face image features and residual kernels of the expression correction model, to obtain the expression-corrected expressionless face image.
[0108] In a possible implementation, the input module 302 is further configured to, for each pixel block in the face image feature, subtract the pixel correction kernel from the pixel block to obtain a first corrected face image feature.
[0109] In a possible implementation, the input module 302 is further configured to:
[0110] After the first corrected face image feature is input into the convolution layer for convolution, a convolutional first corrected face image feature is obtained.
[0111] The convolutional first corrected face image feature is subtracted from the convolution correction kernel to obtain a second corrected face image feature.
[0112] In a possible implementation, the input module 302 is further configured to:
[0113] The expression-corrected expressionless face image is obtained through the following formula:
[0114] F = (1 - W r )E;
[0115] wherein F is the expression-corrected expressionless face image, W r is the residual kernel, and E is the second corrected face image feature.
[0116] In a possible implementation, the apparatus further includes an enhancement module 303.
[0117] The enhancement module 303 is configured to perform adaptive enhancement on the expression-corrected expressionless face image to obtain a final expression-corrected expressionless face image.
[0118] In a possible implementation, the enhancement module 303 is specifically configured to perform adaptive enhancement on the expression-corrected expressionless face image through the following formula to obtain the final expression-corrected expressionless face image: wherein K is the final expression-corrected expressionless face image, F is the expressionless face image before adaptive enhancement, f is an element in F, and the Hamanton vector product is.
[0119] As shown in FIG. 4, Figure 4 the electronic device 400 provided by the embodiment of the present application includes a processor 401, a memory 402, and a bus. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device is running, the processor 401 and the memory 402 communicate through the bus. The processor 401 executes the machine-readable instructions to perform the steps of the expression correction method of the face image as described above.
[0120] Specifically, the memory 402 and the processor 401 can be general memory and processor, which are not specifically limited here, and when the processor 401 runs the computer program stored in the memory 402, the above-mentioned expression correction method of the face image can be executed.
[0121] Corresponding to the expression correction method of the face image, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by the processor, the steps of the expression correction method of the face image are executed.
[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-mentioned system and device can refer to the corresponding process in the method embodiment, which will not be repeated in the present application. In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The above-described device embodiments are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some communication interface, indirect coupling or communication connection between the devices or modules, which can be electrical, mechanical or other forms.
[0123] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.
[0124] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.
[0125] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the information processing method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various storage program codes.
[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A method for correcting facial expressions in a human face image, characterized in that, The facial expression correction method for the facial image includes: Acquire a face image and the facial expression features corresponding to the face image; The facial expression features corresponding to the face image are input into the expression classification model to obtain the expression category corresponding to the face image; The face image is input into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image; wherein, the expression classification model is trained using face expression feature samples and corresponding expression categories; the expression correction model is trained using face image samples and corresponding expressionless face images; The expression-corrected expressionless face image is adaptively enhanced using the following formula to obtain the final expression-corrected expressionless face image. ； Where K is the final expressionless face image after expression correction, F is the expressionless face image before adaptive enhancement, and f is an element in F. It is the Harmanton vector product; The step of inputting the face image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image includes: performing pixel correction on the face image features based on the face image features of the face image and the pixel correction kernel of the expression correction model to obtain a first corrected face image feature; performing convolution correction on the first corrected face image feature based on the first corrected face image feature and the convolution correction kernel of the expression correction model to obtain a second corrected face image feature; and performing feature-level correction on the second corrected face image feature based on the second corrected face image feature and the residual kernel of the expression correction model to obtain the expression-corrected expressionless face image.
2. The facial expression correction method for human face images according to claim 1, characterized in that, The step of performing pixel correction on the facial image features based on the facial image features of the facial image and the pixel correction kernel of the expression correction model to obtain the first corrected facial image features includes: For each pixel block in the facial image feature, the pixel correction kernel is subtracted from the pixel block to obtain the first corrected facial image feature.
3. The facial expression correction method for human face images according to claim 1, characterized in that, The step of performing convolutional correction on the first corrected face image features based on the first corrected face image features and the convolutional correction kernel of the expression correction model to obtain the second corrected face image features includes: After inputting the first corrected face image features into the convolutional layer for convolution, the first corrected face image features after convolution are obtained. The second corrected face image feature is obtained by subtracting the convolution correction kernel from the first corrected face image feature after convolution.
4. The facial expression correction method for human face images according to claim 1, characterized in that, The step of performing feature-level correction on the second modified face image features based on the second modified face image features and the residual kernel of the expression correction model to obtain an expressionless face image after expression correction includes: The expressionless face image after expression correction is obtained using the following formula; ； Where F is the expressionless face image after expression correction, W r is the residual kernel, and E is the second corrected face image feature.
5. A facial expression correction device, characterized in that, The facial expression correction device includes: The acquisition module is used to acquire a face image and the facial expression features corresponding to the face image; The input module is used to input the facial expression features corresponding to the face image into the expression classification model to obtain the expression category corresponding to the face image; The input module is further configured to input the face image into an expression correction model corresponding to the expression category to obtain an expression-corrected expressionless face image; wherein, the expression classification model is trained using face expression feature samples and corresponding expression categories; the expression correction model is trained using face image samples and corresponding expressionless face images; The enhancement module is specifically used to adaptively enhance the expressionless face image after expression correction using the following formula to obtain the final expressionless face image after expression correction. ； Where K is the final expressionless face image after expression correction, F is the expressionless face image before adaptive enhancement, and f is an element in F. It is the Harmanton vector product; Specifically, the input module is used to perform pixel-level correction on the facial image features based on the facial image features of the facial image and the pixel correction kernel of the expression correction model to obtain a first corrected facial image feature; to perform convolution correction on the first corrected facial image feature based on the first corrected facial image feature and the convolution correction kernel of the expression correction model to obtain a second corrected facial image feature; and to perform feature-level correction on the second corrected facial image feature based on the second corrected facial image feature and the residual kernel of the expression correction model to obtain an expressionless facial image after expression correction.
6. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the facial expression correction method for a face image as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the facial expression correction method for a human face image as described in any one of claims 1 to 4.
Citation Information
Patent Citations
Facial expression image synthesis device
CN108230239A
Expression interference resistant face recognition method based on generative adversarial network
CN110276252A