A method and device for recognizing facial expression

The method uses an autoencoder neural network to encode and fuse facial features, improving facial expression recognition accuracy by filtering irrelevant data and reconstructing deep information, addressing environmental interference challenges.

CN112613416BActive Publication Date: 2024-12-03AGRICULTURAL BANK OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011569124.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-26
Publication Date
2024-12-03
Estimated Expiration
2040-12-26

AI Technical Summary

Technical Problem

Existing facial expression recognition methods are prone to interference from environmental factors such as lighting, occlusion, and skin color, leading to inaccurate or failed recognition.

Method used

A method utilizing an autoencoder neural network to encode and fuse texture, geometric, and semantic features from facial images, followed by decoding to enhance recognition accuracy.

Benefits of technology

Improves facial expression recognition accuracy by filtering irrelevant data and reconstructing deep information related to facial expressions, enhancing resistance to environmental interference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112613416B_ABST
    Figure CN112613416B_ABST
Patent Text Reader

Abstract

The present application provides a facial expression recognition method and related devices, which obtain texture features, geometric features and semantic features corresponding to a facial image, and then encode the texture features, geometric features and semantic features respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature. Since the encoding process is a lossy compression of the data, encoding the features is equivalent to filtering the data in the features that are not related to facial expression recognition, thereby reducing the impact of interference information on facial expression recognition. Subsequently, the first feature, the second feature and the third feature are fused to achieve reconstruction of shallow features in three dimensions of the facial image. By decoding the fused features obtained after fusion, the depth information related to facial expression recognition in the facial image is further explored, thereby determining the facial expression corresponding to the facial image according to the decoded fourth feature, thereby improving the recognition of facial expressions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a facial expression recognition method and related devices. Background Art

[0002] Facial expression recognition belongs to the category of emotion recognition, which refers to assigning an emotion category to a given face image, including happiness, sadness, fear, surprise, anger or disgust.

[0003] Facial expression recognition is an important research topic in the field of computer vision. With the development of artificial intelligence research, it has shown broad application prospects in the fields of human interaction, network monitoring and virtual reality. Facial expressions are a reflection of human inner activities. How to accurately recognize facial expressions helps to accurately analyze human emotions, thereby improving the human-computer interaction experience.

[0004] However, related facial expression recognition methods have poor anti-interference capabilities and are easily affected by the environment, such as lighting, occlusion, skin color, face shape, etc., which can easily lead to failure to recognize or incorrect recognition of expressions. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned related technologies, the present application provides a facial expression recognition method and related devices, which reduce the impact of the environment and improve the accuracy of facial expression recognition.

[0006] On the one hand, an embodiment of the present application provides a method for recognizing facial expressions, the method comprising:

[0007] Obtain texture features, geometric features and semantic features corresponding to the face image;

[0008] Encoding the texture feature, the geometric feature, and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature;

[0009] fusing the first feature, the second feature and the third feature to obtain a fused feature corresponding to the face image;

[0010] Decoding the fused feature to obtain a fourth feature corresponding to the face image;

[0011] According to the fourth feature, the facial expression corresponding to the facial image is determined.

[0012] In a possible implementation, the encoding of the texture feature, the geometric feature, and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature includes:

[0013] Encoding the texture feature, the geometric feature and the semantic feature respectively by using an encoding module in an autoencoder neural network model to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature;

[0014] The fusing of the first feature, the second feature and the third feature to obtain a fused feature corresponding to the face image includes:

[0015] Using a fusion module in the autoencoder neural network model to fuse the first feature, the second feature and the third feature, to obtain a fusion feature corresponding to the face image;

[0016] The decoding of the fusion feature to obtain the fourth feature corresponding to the face image includes:

[0017] The fused feature is decoded using a decoding module of the autoencoder neural network model to obtain a fourth feature of the facial image.

[0018] In a possible implementation, the method further includes:

[0019] Acquire a facial image to be identified and a block image corresponding to the facial image; the block image is used to identify the facial parts included in the facial image;

[0020] Acquire the texture features and the geometric features corresponding to the face image according to the block images;

[0021] According to the facial image, a semantic feature corresponding to the facial image is obtained.

[0022] In a possible implementation, the block image is obtained by:

[0023] Positioning feature points of the face image to obtain a plurality of feature points of the face included in the face image;

[0024] According to the positioning marks corresponding to the multiple feature points, the face image is divided to obtain the block images corresponding to the face image.

[0025] In a possible implementation, the autoencoder neural network model is trained according to the following method:

[0026] Obtaining a training sample corresponding to the autoencoder neural network model; the training sample includes a sample face image and a facial expression label corresponding to the sample face image;

[0027] Using the training samples to train the autoencoder neural network model;

[0028] During the training process, the sample facial expression of the sample facial image is determined based on the output of the autoencoder neural network model; and the parameters of the autoencoder neural network model are adjusted according to the sample facial expression and the facial expression label.

[0029] On the other hand, an embodiment of the present application provides a facial expression recognition device, the device comprising an acquisition unit, an encoding unit, a fusion unit, a decoding unit and a determination unit:

[0030] The acquisition unit is used to acquire texture features, geometric features and semantic features corresponding to the face image;

[0031] The encoding unit is used to encode the texture feature, the geometric feature and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature;

[0032] The fusion unit is used to fuse the first feature, the second feature and the third feature to obtain a fusion feature corresponding to the face image;

[0033] The decoding unit is used to decode the fused feature to obtain a fourth feature corresponding to the face image;

[0034] The determining unit is used to determine the facial expression corresponding to the facial image according to the fourth feature.

[0035] In a possible implementation, the encoding unit is used to encode the texture feature, the geometric feature, and the semantic feature respectively using an encoding module in an autoencoder neural network model to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature;

[0036] The fusion unit is used to fuse the first feature, the second feature and the third feature using a fusion module in the autoencoder neural network model to obtain a fusion feature corresponding to the face image;

[0037] The decoding unit is used to decode the fused feature using the decoding module of the autoencoder neural network model to obtain the fourth feature of the face image.

[0038] In a possible implementation, the acquisition unit is further configured to:

[0039] Acquire a facial image to be identified and a block image corresponding to the facial image; the block image is used to identify the facial parts included in the facial image;

[0040] Acquire the texture features and the geometric features corresponding to the face image according to the block images;

[0041] According to the facial image, a semantic feature corresponding to the facial image is obtained.

[0042] In a possible implementation, the device further includes a positioning unit and a dividing unit;

[0043] The positioning unit is used to locate feature points of the face image to obtain a plurality of feature points of the face included in the face image;

[0044] The positioning unit is used to divide the face image according to the positioning identifiers corresponding to the multiple feature points to obtain the block images corresponding to the face image.

[0045] In a possible implementation manner, the device further includes a training unit:

[0046] The training unit is used to obtain training samples corresponding to the autoencoder neural network model; the training samples include sample face images and facial expression labels corresponding to the sample face images;

[0047] Using the training samples to train the autoencoder neural network model;

[0048] During the training process, the sample facial expression of the sample facial image is determined based on the output of the autoencoder neural network model; and the parameters of the autoencoder neural network model are adjusted according to the sample facial expression and the facial expression label.

[0049] It can be seen from the above technical solution that the texture features, geometric features and semantic features corresponding to the face image are obtained, and then the texture features, geometric features and semantic features are encoded respectively to obtain the first feature corresponding to the texture feature, the second feature corresponding to the geometric feature and the third feature corresponding to the semantic feature. Since the encoding process is a lossy compression of the data, encoding the features is equivalent to filtering the data in the features that are not related to facial expression recognition, thereby reducing the influence of interference information on facial expression recognition. Subsequently, the first feature, the second feature and the third feature are fused to achieve the reconstruction of the shallow features of the face image in three dimensions. By decoding the fused features obtained after fusion, the depth information related to facial expression recognition in the face image is further explored, so that the facial expression corresponding to the face image is determined according to the decoded fourth feature, thereby improving the accuracy of facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A schematic diagram of a facial expression recognition method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a facial image preprocessing process provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a face block image provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a texture feature extraction process provided in an embodiment of the present application;

[0055] Figure 5 A flowchart of a semantic feature extraction method provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the structure of an autoencoding neural network model provided in an embodiment of the present application;

[0057] Figure 7 A flowchart of another facial expression recognition method provided in an embodiment of the present application;

[0058] Figure 8 A flowchart of another facial expression recognition method provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of another facial expression recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] The shooting environment of the face image and the state of the subject may affect the recognition accuracy of facial expressions. For example, the subject in the face image is in a weak lighting environment, or the subject is wearing a hat or a mask. In order to improve the recognition accuracy of facial expressions, the embodiments of the present application provide a facial expression recognition method and a related device.

[0062] The facial expression recognition method provided in the present application can be applied to facial expression recognition devices with data processing capabilities, such as terminal devices and servers. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0063] For ease of understanding, the facial expression recognition method provided in the embodiment of the present application is introduced below using a terminal device with an image capture function as a facial expression recognition device.

[0064] See also Figure 1 , Figure 1 A flow chart of a facial expression recognition method provided in an embodiment of the present application. Figure 1 As shown, the facial expression recognition method comprises the following steps:

[0065] S101: Acquire texture features, geometric features, and semantic features corresponding to a face image.

[0066] The facial image can be captured by a camera in real time for the object to be identified, and the facial image corresponding to the object to be identified can be obtained. For example, in a human-computer interaction scenario, the terminal device captures the facial image of the object to be identified through the front camera, and then recognizes the facial expression of the object to be identified based on the facial image. The facial image can also be an image to be identified that is pre-stored in a database, which can be determined according to the actual scenario and is not limited here.

[0067] In related facial expression recognition technologies, the extracted features are relatively simple and cannot fully extract facial expression information, resulting in low efficiency and low accuracy in facial expression recognition.

[0068] In view of this, in the embodiment of the present application, facial images are subjected to expression feature extraction from three different dimensions to obtain texture features, geometric features and semantic features corresponding to the facial images. Expression feature extraction refers to extracting data representing expression features from facial images through relevant algorithms.

[0069] Texture features are used to identify the texture information of the local face in the face image, such as frowns and wrinkles. Geometric features are used to identify the shape information of the face image, such as size and angle. Semantic features are used to identify the attribute information of the face structure included in the face image, such as eyes, nose, mouth, etc., and describe the face image through image information, especially high-level information.

[0070] The above method extracts facial expression information from three different aspects by extracting texture, geometry and semantic features from facial images. Compared with a single feature, facial expression information is utilized more fully and completely, providing rich data for subsequent facial expression recognition based on these three features, thereby improving the recognition of facial expressions.

[0071] Regarding the above-mentioned acquisition of texture features, geometric features and semantic features corresponding to the facial image, in a possible implementation method, the facial image of the object to be identified and the block images corresponding to the facial image are first obtained, and then, feature extraction is performed on the block images to obtain the texture features and geometric features corresponding to the facial image, and feature extraction is performed on the facial image to obtain the semantic features corresponding to the facial image.

[0072] An embodiment of the present application provides a method for dividing a facial image into blocks, namely, locating feature points of the facial image to obtain multiple feature points of the face included in the facial image, and then dividing the facial image according to the positioning identifiers corresponding to each of the multiple feature points to obtain block images corresponding to the facial image.

[0073] In practical applications, the collected face image can be preprocessed first, that is, the face image is converted from RGB space to grayscale space, and then the face image is normalized to obtain a normalized face grayscale image. Then, the active appearance model (AAM) can be used to locate the feature points in the face grayscale image, and each feature point is marked by a positioning identifier. Among them, the number of feature points can be set according to the actual scene, and the positioning identifier can be a digital number, code, etc., which is not limited here. For example, 68 feature points of the face are marked with digital numbers. The specific marking process can be marked according to the structural characteristics of the face. For example, numbers 0 to 16 are used to mark the facial contour, and 17 to 26 are used to mark eyebrows, etc. Then, according to the shape of each part of the face, the above-mentioned multiple marked feature points are connected to form multiple closed irregular polygonal feature blocks as block images.

[0074] For a better understanding, the following Figure 2 , the above image processing process is introduced. Figure 2As shown in FIG. 1 , the image processing process includes a model preparation process and an image segmentation process, wherein the image segmentation process is based on the model preparation process.

[0075] The model preparation phase includes the establishment of the AAM model, the training of the AAM model, and the generation of the AAM model. The image segmentation phase includes AAM model matching, face detection, and model matching. Figure 3 Shows a face image Figure 2 The image shown is a schematic diagram of a block image obtained after image processing.

[0076] The above-mentioned method of locating facial feature points and dividing the facial image into blocks can not only better analyze the local information of the face, but also better conform to the victorious structure of the face, and can better capture the deformation information of each part of the face, laying a good foundation for subsequent feature extraction.

[0077] In the embodiment of the present application, texture features, geometric features and semantic features are all expression feature vectors, and the expression feature vector is a one-dimensional vector composed of data that can effectively represent facial expressions.

[0078] Texture features can be extracted using local binary pattern (LBP), histogram of oriented gradients (HOG) or local directional numeric pattern (LDN). Figure 4 As shown in the figure, the pixel difference-local directional number pattern (PD-LDN) is used to extract the texture features of the block image, and the features are connected and statistically analyzed to form a feature histogram. The vector formed by the histogram data is used as the texture feature vector of this face image.

[0079] Since the positions of facial feature points will change to a certain extent between different expressions, and the degree of position change of feature points is different for different expressions, the position change of feature points will cause the size and shape of each block image to change significantly. Therefore, for the same block image, its moment features between different expressions will also be quite different.

[0080] Therefore, the geometric feature extraction process of the present application can extract n geometric moments from each block image as the geometric features of the block, where n is an integer greater than or equal to 1. In the embodiment of the present application, n=7, that is, the seventh-order moment feature, is used to identify the shape and size changes of each block image. The seventh-order moment feature reflects the difference in geometric information between different expressions and has good recognition and representation.

[0081] The semantic feature extraction process of the present application can first select the key area of ​​facial expression for the face image, and then use the dense scale-invariant feature transform (DSIFT) algorithm to extract the semantic features, that is, perform k-means clustering on the DSIFT features of the key area, generate a dictionary, and obtain a statistical histogram as the semantic feature of the face image, such as Figure 5 shown.

[0082] It should be noted that the above is only a possible implementation method for obtaining texture features, geometric features and semantic features corresponding to a facial image provided in an embodiment of the present application. In actual applications, it can be set according to the actual scene and is not limited here.

[0083] S102: Encode the texture feature, the geometric feature, and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature.

[0084] Since most of the related facial expression recognition technologies directly use the features extracted from facial images to recognize facial expressions, the feature recognition degree is low.

[0085] To this end, the present application provides a possible implementation method, namely, deeply fusing the texture features, geometric features and semantic features obtained in the above S101, so as to improve the accuracy of facial expression recognition.

[0086] First, the texture feature, geometric feature and semantic feature can be encoded respectively to obtain the first feature corresponding to the texture feature, the second feature corresponding to the geometric feature and the third feature corresponding to the semantic feature. Since the encoding process is a lossy compression of the data, encoding the feature is equivalent to filtering the data in the feature that is not related to facial expression recognition, reducing the impact of interference information on facial expression recognition.

[0087] S103: Fusing the first feature, the second feature, and the third feature to obtain a fused feature corresponding to the face image.

[0088] Then, the first feature, the second feature and the third feature are fused to obtain a fused feature corresponding to the face image, where the fused feature identifies texture information, geometric information and semantic information included in the face image.

[0089] Since the fusion feature integrates the features of three different dimensions of the facial image and complements the advantages and disadvantages of different features, the subsequent recognition of facial expressions based on the fusion feature can maintain a good expression recognition rate in different scenarios and improve the performance of resisting interference from external factors.

[0090] S104: Decode the fused feature to obtain a fourth feature corresponding to the face image.

[0091] Since the face image is encoded and compressed in S102, the fused feature needs to be decoded accordingly to obtain the fourth feature corresponding to the face image.

[0092] The above process of encoding, fusing and decoding features realizes the deep fusion of shallow features, that is, the feature deep fusion process, which strengthens the connection between texture features, geometric features and semantic features in facial images, enriches the data based on face recognition, and provides deeper information for the subsequent use of the decoded fourth feature to recognize facial expressions.

[0093] S105: Determine the facial expression corresponding to the facial image according to the fourth feature.

[0094] In practical applications, the facial expression corresponding to the face image, such as happiness, anger, sadness, etc., can be determined by using a classifier based on the fourth feature.

[0095] The above three features after fusion encoding have been realized to reconstruct the shallow features of the face image in three dimensions. By decoding the fused features obtained after fusion, the deep information related to facial expression recognition in the face image is further discovered, so that the facial expression corresponding to the face image is determined according to the decoded fourth feature, thereby improving the accuracy of facial expression recognition.

[0096] For the above process of recognizing facial expressions using texture features, geometric features and semantic features, an embodiment of the present application provides a three-channel autoencoding neural network model, including an encoding module, a fusion module and a decoding module.

[0097] For a better understanding, the following Figure 6 For introduction. Figure 6 As shown, the autoencoding neural network model includes three layers, namely, input layer, hidden layer and output layer.

[0098] During the application process, the input texture feature x p , geometric features x g and semantic feature x b , use the encoding module to encode them respectively and get the first feature h p , the second feature h g and the third characteristic h b Then, the fusion module is used to transform the first feature h p , the second feature h g and the third characteristic h bThe fusion feature is obtained by fusion, and the fusion feature is decoded by the decoding module to obtain the fourth feature h f , and input into the classifier softmax to determine the expression category corresponding to the face image. In this embodiment, the encoding module is a matrix used to encode the features, such as Figure 6 The encoding matrix W shown p , W g and W b ; The fusion module and the decoding module are matrices used to fuse and decode features, such as Figure 6 The fused decoding matrix W shown f .

[0099] As can be seen from the above, the facial expression recognition method provided in the embodiment of the present application is implemented based on artificial intelligence technology, especially deep learning in artificial intelligence. The initial model parameters of the autoencoder neural network model can be adjusted using training samples.

[0100] During the application process, a training sample corresponding to the autoencoder neural network model can be obtained, and the training sample includes a sample face image and a facial expression label corresponding to the sample face image. The facial expression label identifies the expression category of the sample face image. Then, the autoencoder neural network model is trained using the training sample.

[0101] During the training process, the sample facial expression of the sample face image is determined based on the output of the autoencoder neural network model, and then the parameters of the autoencoder neural network model are adjusted according to the loss between the sample facial expression and the facial expression label.

[0102] The facial expression recognition method provided in the above embodiment obtains the texture features, geometric features and semantic features corresponding to the facial image, and then encodes the texture features, geometric features and semantic features respectively to obtain the first feature corresponding to the texture feature, the second feature corresponding to the geometric feature and the third feature corresponding to the semantic feature. Since the encoding process is a lossy compression of the data, encoding the features is equivalent to filtering the data in the features that are not related to facial expression recognition, thereby reducing the influence of interference information on facial expression recognition. Subsequently, the first feature, the second feature and the third feature are fused to achieve the reconstruction of the shallow features of the facial image in three dimensions. By decoding the fused features obtained after fusion, the depth information related to facial expression recognition in the facial image is further explored, thereby determining the facial expression corresponding to the facial image according to the decoded fourth feature, thereby improving the accuracy of facial expression recognition.

[0103] In order to better understand the facial expression recognition method provided by the above embodiment, Figure 7 and Figure 8 , the application process of facial expression recognition method is introduced.

[0104] First, the face image, that is, the expression image, is preprocessed to obtain a block image. Then, texture feature extraction and geometric feature extraction are performed on the block image, and semantic feature extraction is performed on the face image, that is, Figure 8 The PD-LDN features, seventh-order moments, and semantic features are extracted using the bag-of-words model. Then, the texture features, geometric features, and semantic features are encoded and fused respectively, and the fused features are decoded, that is, feature deep fusion processing. Finally, the classifier is used to determine the expression category of the face image, whether it is happy, surprised, or sad, etc.

[0105] The above method extracts three features of facial images and deeply fuses these three shallow features, which reduces the interference with facial expression information while increasing the deep information of the connection between features of different dimensions, further improving the anti-interference ability and facial expression recognition.

[0106] In view of the facial expression recognition method provided in the above embodiment, the embodiment of the present application also provides a facial expression recognition device.

[0107] See also Fig. 9 , Fig. 9 A facial expression recognition device is provided in an embodiment of the present application. Fig. 9 As shown, the facial expression recognition device 900 includes an acquisition unit 901, an encoding unit 902, a fusion unit 903, a decoding unit 904 and a determination unit 905:

[0108] The acquisition unit 901 is used to acquire texture features, geometric features and semantic features corresponding to the face image;

[0109] The encoding unit 902 is used to encode the texture feature, the geometric feature and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature;

[0110] The fusion unit 903 is used to fuse the first feature, the second feature and the third feature to obtain a fusion feature corresponding to the face image;

[0111] The decoding unit 904 is used to decode the fused feature to obtain a fourth feature corresponding to the face image;

[0112] The determining unit 905 is used to determine the facial expression corresponding to the facial image according to the fourth feature.

[0113] In a possible implementation, the encoding unit 902 is used to encode the texture feature, the geometric feature, and the semantic feature respectively using an encoding module in an autoencoder neural network model to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature;

[0114] The fusion unit 903 is used to fuse the first feature, the second feature and the third feature by using the fusion module in the autoencoder neural network model to obtain a fusion feature corresponding to the face image;

[0115] The decoding unit 904 is used to decode the fused feature using the decoding module of the autoencoder neural network model to obtain the fourth feature of the face image.

[0116] In a possible implementation, the acquisition unit 901 is further configured to:

[0117] Acquire a facial image to be identified and a block image corresponding to the facial image; the block image is used to identify the facial parts included in the facial image;

[0118] Acquire the texture features and the geometric features corresponding to the face image according to the block images;

[0119] According to the facial image, a semantic feature corresponding to the facial image is obtained.

[0120] In a possible implementation, the device further includes a positioning unit and a dividing unit;

[0121] The positioning unit is used to locate feature points of the face image to obtain a plurality of feature points of the face included in the face image;

[0122] The positioning unit is used to divide the face image according to the positioning identifiers corresponding to the multiple feature points to obtain the block images corresponding to the face image.

[0123] In a possible implementation manner, the device further includes a training unit:

[0124] The training unit is used to obtain training samples corresponding to the autoencoder neural network model; the training samples include sample face images and facial expression labels corresponding to the sample face images;

[0125] Using the training samples to train the autoencoder neural network model;

[0126] During the training process, the sample facial expression of the sample facial image is determined based on the output of the autoencoder neural network model; and the parameters of the autoencoder neural network model are adjusted according to the sample facial expression and the facial expression label.

[0127] The facial expression recognition device provided in the above embodiment obtains the texture features, geometric features and semantic features corresponding to the facial image, and then encodes the texture features, geometric features and semantic features respectively to obtain the first feature corresponding to the texture feature, the second feature corresponding to the geometric feature and the third feature corresponding to the semantic feature. Since the encoding process is a lossy compression of the data, encoding the features is equivalent to filtering the data in the features that are not related to facial expression recognition, thereby reducing the influence of interference information on facial expression recognition. Then, the first feature, the second feature and the third feature are fused to achieve the reconstruction of the shallow features of the facial image in three dimensions. By decoding the fused features obtained after the fusion, the depth information related to facial expression recognition in the facial image is further explored, so that the facial expression corresponding to the facial image is determined according to the decoded fourth feature, thereby improving the accuracy of facial expression recognition.

[0128] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0129] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for recognizing facial expressions, characterized in that: The method comprises: Acquire texture features, geometric features, and semantic features corresponding to the face image, wherein the semantic features are used to identify attribute information of the face structure included in the face image; Encoding the texture feature, the geometric feature, and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature, and a third feature corresponding to the semantic feature; fusing the first feature, the second feature and the third feature to obtain a fused feature corresponding to the face image; Decoding the fused feature to obtain a fourth feature corresponding to the face image; Determining the facial expression corresponding to the facial image according to the fourth feature; The obtaining of texture features, geometric features and semantic features corresponding to the face image includes: Acquire a facial image to be identified and a block image corresponding to the facial image; the block image is used to identify the facial parts included in the facial image; According to the block image, the texture feature and the geometric feature corresponding to the face image are obtained, where the geometric feature is n geometric moments of the block image, where n is an integer greater than or equal to 1; According to the facial image, obtaining semantic features corresponding to the facial image; The acquiring, according to the face image, a semantic feature corresponding to the face image, comprises: Selecting a key facial expression area of ​​the facial image; The semantic features of the key areas of facial expression are extracted using a dense scale-invariant feature conversion algorithm to obtain semantic features corresponding to the facial image.

2. The method according to claim 1, characterized in that The encoding of the texture feature, the geometric feature and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature comprises: Encoding the texture feature, the geometric feature and the semantic feature respectively by using an encoding module in an autoencoder neural network model to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature; The fusing of the first feature, the second feature and the third feature to obtain a fused feature corresponding to the face image includes: Using a fusion module in the autoencoder neural network model to fuse the first feature, the second feature and the third feature, to obtain a fusion feature corresponding to the face image; The decoding of the fusion feature to obtain the fourth feature corresponding to the face image includes: The fused feature is decoded using a decoding module of the autoencoder neural network model to obtain a fourth feature of the facial image.

3. The method according to claim 1, characterized in that The block image is obtained by: Positioning feature points of the face image to obtain a plurality of feature points of the face included in the face image; According to the positioning marks corresponding to the multiple feature points, the face image is divided to obtain the block images corresponding to the face image.

4. The method according to claim 2, characterized in that: The autoencoder neural network model is trained according to the following method: Obtaining a training sample corresponding to the autoencoder neural network model; the training sample includes a sample face image and a facial expression label corresponding to the sample face image; Using the training samples to train the autoencoder neural network model; During the training process, the sample facial expression of the sample facial image is determined based on the output of the autoencoder neural network model; and the parameters of the autoencoder neural network model are adjusted according to the sample facial expression and the facial expression label.

5. A facial expression recognition device, characterized in that: The device includes an acquisition unit, an encoding unit, a fusion unit, a decoding unit and a determination unit: The acquisition unit is used to acquire texture features, geometric features and semantic features corresponding to the face image, wherein the semantic features are used to identify attribute information of the face structure included in the face image; The encoding unit is used to encode the texture feature, the geometric feature and the semantic feature respectively to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature; The fusion unit is used to fuse the first feature, the second feature and the third feature to obtain a fusion feature corresponding to the face image; The decoding unit is used to decode the fused feature to obtain a fourth feature corresponding to the face image; The determining unit is used to determine the facial expression corresponding to the facial image according to the fourth feature; The acquisition unit is further used for: Acquire a facial image to be identified and a block image corresponding to the facial image; the block image is used to identify the facial parts included in the facial image; According to the block image, the texture feature and the geometric feature corresponding to the face image are obtained, where the geometric feature is n geometric moments of the block image, where n is an integer greater than or equal to 1; According to the facial image, obtaining semantic features corresponding to the facial image; When the acquisition unit acquires the semantic feature corresponding to the face image according to the face image, the acquisition unit is specifically used to: Selecting a key facial expression area of ​​the facial image; The semantic features of the key areas of facial expression are extracted using a dense scale-invariant feature conversion algorithm to obtain semantic features corresponding to the facial image.

6. The device according to claim 5, characterized in that The encoding unit is used to encode the texture feature, the geometric feature and the semantic feature respectively by using the encoding module in the autoencoder neural network model to obtain a first feature corresponding to the texture feature, a second feature corresponding to the geometric feature and a third feature corresponding to the semantic feature; The fusion unit is used to fuse the first feature, the second feature and the third feature using a fusion module in the autoencoder neural network model to obtain a fusion feature corresponding to the face image; The decoding unit is used to decode the fused feature using the decoding module of the autoencoder neural network model to obtain the fourth feature of the face image.

7. The device according to claim 5, characterized in that The device also includes a positioning unit and a dividing unit; The positioning unit is used to locate feature points of the face image to obtain a plurality of feature points of the face included in the face image; The positioning unit is used to divide the face image according to the positioning identifiers corresponding to the multiple feature points to obtain the block images corresponding to the face image.

8. The device according to claim 6, characterized in that The device also includes a training unit: The training unit is used to obtain training samples corresponding to the autoencoder neural network model; the training samples include sample face images and facial expression labels corresponding to the sample face images; Using the training samples to train the autoencoder neural network model; During the training process, the sample facial expression of the sample facial image is determined based on the output of the autoencoder neural network model; and the parameters of the autoencoder neural network model are adjusted according to the sample facial expression and the facial expression label.