A method and device for recognizing facial expressions
By extracting the amplitude and phase characteristics of the target frame image in the video image, and combining the GRAY-YCrCb color model and mixed filtering technology, the problem of low accuracy of facial expression recognition in multiple perspectives is solved, significantly improving the accuracy of expression recognition sex.
Patent Information
- Application Number
- CN202410156471.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-02-02
AI Technical Summary
In the prior art, under multi-view angles, there are large errors in texture feature acquisition of face images, resulting in low accuracy in facial expression recognition.
By obtaining the target frame image in the video image, extracting its amplitude and phase characteristics, and inputting it into the neural network for recognition, the GRAY-YCrCb color model and hybrid filtering technology are used to significantly improve the accuracy of expression recognition.
It significantly improves the accuracy of facial expression recognition, enhances the significance and recognition accuracy of expression characteristics.
Smart Images

Figure CN118038520B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the technical field of image processing, and in particular, to a method and device for recognizing facial expressions. Background Art
[0002] Automatic detection of facial expressions has become an extremely critical research area, involving multiple fields such as computer vision, machine learning, and behavioral science. This technology is not only widely used in multiple fields such as security, human-computer interaction, intelligent driving, and medical care, but also has made significant progress in machines judging individual emotions and even inferring their intentions through facial expressions and making corresponding responses. For example, in terms of preventing emotional terrorists and criminals from entering a building, when the machine automatically recognizes their expressions, it will promptly notify the staff to take corresponding measures. In the future smart companion, the machine will adjust its dialogue mode according to individual facial expressions and provide different solutions. In driving scenarios, the machine can recognize the driver's expression, judge the driver's psychological state based on this, and provide corresponding safety voice prompts to prevent different degrees of reactions, such as fatigue driving or overexcitement. In unattended wards, the machine can judge the patient's physical condition by capturing changes in the patient's facial expressions, providing assistance in verifying the effectiveness of medical solutions.
[0003] Existing technologies are very sensitive to facial posture and orientation. Especially in multi-viewing situations, there are large errors in the texture features of the collected facial images, resulting in low accuracy in facial expression recognition. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for recognizing facial expressions, so as to at least solve the problem that when there are large errors in the texture features of the collected facial images, the accuracy of facial expression recognition is low.
[0005] According to an embodiment of the present invention, a method for recognizing facial expressions is provided, comprising:
[0006] Acquire a video image containing a face image;
[0007] Acquire a target frame image in the video image, wherein the feature information of the target frame image is greater than the feature information of other frame images, and the other frame images are images other than the target frame image in the video image;
[0008] Extracting features of the target frame image to obtain amplitude features and phase features;
[0009] The amplitude feature and the phase feature are input into a neural network to obtain an expression recognition result.
[0010] According to one embodiment of the present invention, there is provided a facial expression recognition device, comprising:
[0011] A first acquisition module, used to acquire a video image containing a face image;
[0012] A second acquisition module is used to acquire a target frame image from the video image, wherein the feature information of the target frame image is more than the feature information of other frame images, and the other frame images are images other than the target frame image in the video image;
[0013] An extraction module, used for extracting features from the target frame image to obtain amplitude features and phase features;
[0014] The result output module is used to input the amplitude feature and the phase feature into a neural network to obtain an expression recognition result.
[0015] According to another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps in any embodiment of the above-mentioned facial expression recognition method when running.
[0016] According to another embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any embodiment of the above-mentioned facial expression recognition method.
[0017] Through the present invention, by extracting the target frame in the video image, the expression features of the obtained face image can be made more prominent, and expression recognition based on the face image can improve the accuracy of expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a hardware structure diagram of a mobile terminal of a method for recognizing facial expressions according to an embodiment of the present invention;
[0019] Figure 2 is a flow chart of a method for recognizing facial expressions according to an embodiment of the present invention;
[0020] Figure 3 This is a comparison diagram before and after GRAY-YCrCb processing of a specific embodiment of the present invention;
[0021] Figure 4 This is a comparison diagram before and after BGR conversion to GRAY in a specific embodiment of the present invention;
[0022] Figure 5is a before-and-after comparison diagram of removing wrinkles by the GRAY-YCrCb method according to a specific embodiment of the present invention;
[0023] Figure 6 is a curve diagram of the difference of grayscale values of an image according to a specific embodiment of the present invention;
[0024] Figure 7 is an amplitude feature map of a face image according to a specific embodiment of the present invention;
[0025] Figure 8 is a phase feature map of a face image of a specific embodiment of the present invention;
[0026] Fig. 9 is a dual-channel GCN model diagram of a specific embodiment of the present invention;
[0027] Fig.10 is a maxout-GCN model structure diagram of a specific embodiment of the present invention;
[0028] Fig.11 is a flow chart of a method for recognizing facial expressions according to a specific embodiment of the present invention;
[0029] Fig.12 is a DNN model diagram of the present invention;
[0030] Fig.13 It is the VGG16 model diagram of the present invention;
[0031] Fig.14 It is a device diagram of a method for recognizing facial expressions according to an embodiment of the present invention. DETAILED DESCRIPTION
[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0034] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for recognizing facial expressions according to an embodiment of the present invention. Figure 1As shown, the mobile terminal may include one or more (only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a method for recognizing facial expressions in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0036] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0037] This embodiment provides a method for recognizing a face image, wherein: Figure 2 is a flow chart of a method for recognizing a face image according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:
[0038] Step S201: Acquire a video image containing a face image.
[0039] The video image may be a video image that needs to be recognized for facial expression. For example, a robot needs to obtain facial expressions of a person so as to have an intelligent conversation based on the expressions. The robot may collect video images of the person in real time.
[0040] After acquiring the video image of the face image, the image can be preprocessed, including graying the facial expression, locating the face, cropping, etc.
[0041] Step S202: acquiring a target frame image in the video image, wherein the target frame image has more feature information than other frame images, and the other frame images are images other than the target frame image in the video image.
[0042] In the acquired video image, since the position and expression of the face will change with the acquisition time, a target frame image with more features of the face expression can be acquired in the video image. Specifically, a target frame image with more features than other frame images can be acquired in the video image.
[0043] In a specific implementation, the collected video may be divided into multiple sequences, each of which may contain different facial expressions, so as to obtain a target frame image in each sequence. For example, when a facial image changes from a neutral expression to a surprised expression, it is a sequence frame image.
[0044] Optionally, acquiring a target frame image in the video image includes:
[0045] Acquire a first image in which the facial expression of the human face is a neutral expression in the video image;
[0046] determining a grayscale value of the first image;
[0047] The target frame image is obtained in the video image, the difference between the grayscale value of the target frame image and the grayscale value of the first image being within a first preset range.
[0048] The expressions of human face images usually include: anger, contempt, disgust, fear, happiness, sadness, surprise, neutral expression, etc. Among them, the neutral expression can be understood as the expression of the human face in a calm state.
[0049] A first image with a neutral facial expression is obtained in a video image. In a specific implementation, a sequence frame image of a facial image can be obtained, and a facial image with a neutral expression is obtained in the sequence frame image. When the facial expression gradually changes from a calm neutral state to a surprised state, the first frame image in the sequence frame image can be used as the first image; when the acquired facial expression image gradually changes from a surprised state to a calm state, the last frame image in the sequence frame can be used as the first image. When there are multiple facial images with neutral expressions, an image with higher pixels, clearer images, and obvious features of the facial image can be obtained from the multiple neutral expression images as the first image. Then, the grayscale value of the first image is calculated. Specifically, the first image is first converted into a grayscale image, and then the grayscale value of the pixel point of the image is calculated to obtain the grayscale value of the first image. In order to eliminate the noise of the facial image, the first image can also be first converted into a grayscale image, and after mixed filtering, the grayscale value of the image is calculated by GRAY-YCrCb to obtain the grayscale value of the first image.
[0050] In addition, the grayscale value of each image in the video image except the first image is obtained. When the image in the video image except the first image includes multiple images, the grayscale value of the first image is subtracted from the grayscale value of the multiple images in turn to obtain the difference of multiple grayscale values. The target frame image corresponding to the difference within the first preset range is obtained from the difference of multiple grayscale values, wherein the first preset range can be a difference greater than a predetermined value, or a difference at the first few positions after the difference sizes are arranged in order from large to small, so as to determine the corresponding target frame image according to the obtained difference.
[0051] When the difference between the grayscale value of the image and the grayscale value of the neutral expression image is larger, the features of the face image are more prominent. By the above method, the expression features of the obtained face image can be made more prominent, and expression recognition based on the face image can improve the accuracy of expression recognition.
[0052] Optionally, before acquiring, in the video image, the target frame image whose grayscale value difference with the grayscale value of the first image is within a first preset range, the method further includes:
[0053] Acquire a grayscale value of a second image in the video image, where the second image is an image in the video image other than the first image, and the second image includes at least one frame of image;
[0054] Converting the grayscale value of the first image and the grayscale value of the second image into color space values respectively, to obtain the first image and the second image containing facial contour features of the face image;
[0055] Acquire a first grayscale value based on the first image including the facial contour features of the facial image, and acquire a second grayscale value based on the second image including the facial contour features of the facial image;
[0056] The step of obtaining the target frame image in the video image, the target frame image having a grayscale value whose difference with the grayscale value of the first image is within a first preset range, comprises:
[0057] Acquire a target difference value where the difference between the second grayscale value and the first grayscale value is within the first preset range;
[0058] Determine the target frame image corresponding to the target difference value.
[0059] In order to make the acquired expression features more prominent, when acquiring the difference in the grayscale values of the images, all images in the video image (including the first image and the second image) can be converted first and then the grayscale values can be calculated. Through this conversion process, the first image and the second image contain more prominent facial contour features of the face image.
[0060] In order to achieve the above effects, the present invention adopts a conversion method of GRAY-YCrCb. YCrCb is a commonly used color model for skin color detection. Its spatial composition principle is similar to the human visual perception process, and the clustering effect of skin color points is best in this space. It can realize the separation processing of brightness and chromaticity, and better limit the distribution area of skin color. YCrCb space is also often used for face detection, because most images are based on BGR space. In BGR space, brightness has a great influence on the skin color of the face, so it is a challenging task to separate skin color points from non-skin color points. If the BGR space is converted to YCrCb space, the influence of brightness can be ignored, because the space is less affected by brightness, and the skin color will produce a good clustering effect, which is conducive to highlighting facial features.
[0061] Taking the processing of the second image as an example, a grayscale image of the second image is obtained, and the grayscale value of the second image is converted into a color space value, that is, the color space value of the three channels of blue, green and red, also known as a BGR value. In other words, the grayscale value of the grayscale image is used as the value of the three channels of YCrCb, that is, Y=Cr=Cb=GRAY, and then the second image is converted from YCrCb to a BGR image. Specifically, the conversion can be performed through the following conversion formula:
[0062]
[0063] The BGR value, that is, the color space value of the three channels of blue, green, and red, is obtained through the above conversion formula, and the second image containing significant facial contour features is obtained, such as Figure 3The face image on the left in the figure contains facial features such as wrinkles. After conversion through the above formula, the redundant features such as wrinkles are removed from the obtained face image, making the difference between skin color and non-skin color more significant, the contours of facial features more prominent, and the features of facial expression images more obvious.
[0064] Based on the above conversion, a second image with prominent facial contour features is obtained, and the grayscale value of the second image, that is, the second grayscale value, is obtained. In other words, the BGR value is converted to grayscale to obtain the grayscale value of the pixel point. Specifically, the conversion can be performed using the following formula:
[0065] Gray=0.299·R+0.587·G+0.114·B
[0066] like Figure 4 As shown in the figure, the left side is the face image before BGR is converted to grayscale, and the right side is the face image after BGR is converted to grayscale. It can be seen that after the conversion, the changes in facial expressions and contours are deepened, making the facial expression features clearer and improving the accuracy of subsequent facial expression recognition.
[0067] In order to further reduce the influence of noise on facial features, preferably, the present invention adopts a hybrid filtering method (median filtering and Gaussian filtering combined) and the GRAY-YCrCb method in the above embodiment to extract the key frame of the facial expression image. That is to say, the first image and the second image are processed by the hybrid filtering method of median filtering and Gaussian filtering, and then by the GRAY-YCrCb method. Since median filtering and Gaussian filtering can effectively protect the edge information of the image, combining these two filters can further reduce the influence of noise on the features, and can effectively protect the feature points and edge information of the image.
[0068] like Figure 5 As shown in the figure, the face image is sequentially subjected to Gaussian filtering (a on the left), GRAY-YCrCb conversion (b in the figure), and BGR conversion to GRAY (c on the right). As can be seen from the figure, in Figure a, the salt and pepper noise is eliminated after median filtering. The facial expression after median filtering is then Gaussian filtered to eliminate Gaussian noise and retain the overall grayscale distribution characteristics of the image. Further through GRAY-YCrCb conversion, Figures b and c are obtained. At this time, wrinkles are further eliminated on the basis of Figure a, and the features of the face image are significant. The feature information of facial expressions can be enhanced, and the unique information of the face itself, such as wrinkles, can be filtered, thereby improving the accuracy of subsequent facial expression recognition. Through the above-mentioned GRAY-YCrCb conversion method, the influence of skin color and wrinkles on facial expression features can be effectively reduced, and the efficiency and effect of key frame extraction can be improved.
[0069] Similarly, the first image is converted by the method combining the above hybrid filtering with GRAY-YCrCb. That is, the facial image is firstly subjected to face positioning and grayscale preprocessing, and after hybrid filtering, the values of the grayscale image are respectively used as the three-channel values of YCrCb, and then the facial expression is converted from YCrCb to BGR, so that the contours of the facial features are more prominent and the features of the facial expression image are more obvious. Then, the grayscale value (i.e., the first grayscale value) of the facial expression after GRAY-YCrCb processing is calculated, that is, BGR is converted to grayscale to obtain the grayscale value of the pixel point, thereby deepening the changes in facial expressions and contours, making the features clearer, and facilitating the improvement of the accuracy of subsequent facial expression recognition.
[0070] It should be noted that the present invention does not limit the order of the grayscale value conversion process of the first image and the second grayscale value conversion process.
[0071] The difference is calculated based on the first grayscale value and the second grayscale value. When the second grayscale value includes multiple grayscale values, a target difference is obtained in a first preset range. The first preset range may be a value with a larger difference. Based on the target frame image corresponding to the larger difference, a more significant facial expression can be obtained, which can improve the accuracy of expression recognition.
[0072] Optionally, the video image is a sequence frame image of the face image, and the first image is a first frame image in the sequence frame image; and obtaining a target difference value in which the difference between the second grayscale value and the first grayscale value is within the first preset range includes:
[0073] According to the order in which the sequence frame images are arranged, sequentially acquiring at least two differences between the second grayscale value and the first grayscale value;
[0074] A target difference value arranged within a second preset range is obtained from the at least two difference values.
[0075] In this embodiment, the sequence frame images are sorted according to the order in which the images are taken. When multiple second images are acquired, multiple corresponding second grayscale values can be acquired, thereby obtaining multiple difference values (i.e., at least two difference values). According to the order in which the sequence frame images are arranged, the multiple difference values are arranged to obtain a target difference value arranged within a second preset range, so that the obtained difference value is larger. The second preset range is, for example, the difference values that are in the first few positions in the arrangement order.
[0076] Taking the facial image from a calm neutral expression to a surprised expression as an example, a sequence of facial expressions can be obtained. Figure 6As shown in the figure, the horizontal axis Order number represents the order number of the sequence frame, and the vertical axis Difference represents the difference between the grayscale value of the N-frame image and the grayscale value of the first frame image (neutral expression image). Obtain the difference between the grayscale value of each frame image in the N-frame image and the grayscale value of the first image (neutral expression), and obtain the following Figure 6 The four coordinate diagrams shown in . As can be seen from the figure, as the sequence number of the image increases, the difference in grayscale value increases, so the difference of the sequence number located in the last image can be obtained, that is, the target difference arranged in the second preset range is obtained, so as to obtain an image with significant facial expression features. For example, the three differences with the largest differences among the last five differences are obtained, so as to obtain the target frame images corresponding to these three differences, that is, the three facial expression images with the largest differences are extracted as key frames. It can be seen that when the facial image changes from a neutral expression in a calm state to a surprised expression, the image with the image sequence number arranged in the last few can be obtained as the target frame image, and the difference obtained at this time is relatively large; when the expression of the facial image is gradually changed from a surprised state to a calm state, the image arranged in the first few can be obtained as the target frame image, and the difference obtained at this time is relatively large, the difference in facial expression is relatively large, and the expression features are obvious.
[0077] However, in the sequence frame diagram, there may be a situation where the difference curve reaches a peak in the first few frames, such as Figure 6 For example, in c and d, it is necessary to eliminate the peaks in the first few frames to make the curve conform to the trend of slow growth, and then obtain the images with larger difference in the last few frames.
[0078] The key frame images obtained in the above manner can obtain richer facial expression features and improve the accuracy of facial expression recognition.
[0079] Step S203: extract features from the target frame image to obtain amplitude features and phase features.
[0080] After acquiring the target frame image, Gabor wavelet transform is used to obtain the amplitude and phase features.
[0081] Among them, the kernel function of the two-dimensional Gabor wavelet is defined as follows:
[0082]
[0083] Where u and v represent the direction and size of the Gabor wavelet kernel, respectively. Z = (x, y) represents the image coordinates of a given position. Used to compensate for the energy spectrum attenuation determined by frequency.
[0084] is a Gaussian envelope function that limits the range of the oscillating function by windowing, so it is only valid locally. is an oscillating function, which is a complex number expressed in exponential form and is triangular in actual operation. The two-dimensional Gabor filter can actually be written in complex form, with its real and imaginary parts expressed as:
[0085]
[0086]
[0087] When u and v take different values, a group of Gabor filters with different sizes and directions can be obtained. Since the Gabor wavelet kernel is symmetrical about the origin center, the wavelet kernel should be selected to cover the entire upper half of the cycle. The selection of the wavelet kernel scale should be able to extract the features of the expression as comprehensively as possible. Too little will make it difficult to extract useful information, and too much may lead to information redundancy. The essence of wavelet is multi-resolution analysis, which can extract the features of the original image at different frequencies. The present invention only extracts the features of the image, so a limited number of frequencies are required. At this time, the Gabor wavelet can be regarded as a group of bandpass filters with different frequencies.
[0088] The Gabor features of the image are obtained by convolving the original image with the Gabor wavelet kernel. Assuming that I(x,y) is the gray value set at point (x,y), the Gabor wavelet transform is defined as:
[0089] G u,v (x, y) = I (x, y) * Ψ u,v (x, y)
[0090] G u,v The real and imaginary parts of (x, y) are Re(G) and Im(G) respectively, and the amplitude is expressed as:
[0091]
[0092] The amplitude features of the face image are extracted by the above formula, and the following is obtained: Figure 7 Amplitude features of the facial expression images shown.
[0093] Based on Ψ u,v The real and imaginary parts of (z) are calculated by calculating the four-quadrant inverse tangent of the ratio of the imaginary part to the real part to obtain the phase characteristics of the Gabor wavelet transform. The value range of the traditional inverse tangent function is When Re(G) and Im(G) are both positive or negative or one positive and one negative, the quadrants they are in cannot be distinguished based only on their ratio. The four-quadrant inverse tangent function can distinguish the above two cases by extending the value range to (-π,π). In order to obtain the phase characteristic, the calculation formula is:
[0094]
[0095] According to the above formula, the phase characteristic of Gabor wavelet transform is calculated, and the result is an integer between 0 and 255, which is used to represent the gray value. Figure 8 Phase features of the facial expression images shown.
[0096] The amplitude and phase features extracted through the above process can be used as features of facial expression images in the next training and classification steps.
[0097] Step S204: input the amplitude feature and the phase feature into a neural network to obtain an expression recognition result.
[0098] The present invention designs and constructs a dual-channel neural network model structure, and inputs amplitude features and phase features into the dual channels of the neural network for feature extraction to obtain expression recognition results, such as facial expressions of surprise or anger.
[0099] Optionally, inputting the amplitude feature and the phase feature into a neural network to obtain an expression recognition result includes:
[0100] Inputting the amplitude feature and the phase feature into a dual-channel neural network model to obtain an expression recognition result;
[0101] Among them, one channel of the dual-channel neural network model is used to obtain local features of the face image, and the other channel of the dual-channel neural network model is used to fuse the acquired facial features of the face image to obtain facial expression features.
[0102] The present invention constructs a dual-channel neural network model structure, such as Fig. 9 As shown, it includes two graph convolutional layers, two pooling layers and two fully connected layers.
[0103] Due to the existence of nonlinear activation functions, the GCN model can abstract images and extract features very well. In facial expression recognition, the key points of the face or the pixels in the expression image can be regarded as nodes in the graph, and the relationship between them can be considered as the edges of the graph. Through the modeling of this graph structure, GCN can capture the correlation information between nodes, so as to understand the characteristics of facial expressions more comprehensively. The activation function plays a very important role in the training and classification of GCN, mainly in the following aspects:
[0104] (1) Introducing nonlinearity: The activation function introduces nonlinearity, which enables GCN to learn more complex models. Without the activation function, stacking multiple graph convolutional layers is still equivalent to a linear transformation, which limits the network's expressive power. Nonlinear activation functions (such as ReLU, Sigmoid, etc.) help the network learn more complex feature mappings and improve the ability to model complex data patterns.
[0105] (2) Preventing gradient vanishing and exploding: In deep networks, stacking linear transformations can easily lead to gradient vanishing or gradient exploding problems. Activation functions can introduce nonlinearity in each layer, which helps alleviate these problems and makes gradients easier to propagate.
[0106] (3) Enhance the expressive power of the model: Different activation functions have different adaptability to different data distributions and tasks. Selecting a suitable activation function helps enhance the expressive power of the model and make it more adaptable to specific task requirements.
[0107] Since maxout is an activation function, compared with other activation functions, its fitting ability is very strong and can fit any convex function. The two-layer maxout activation function can fit not only convex functions but also non-convex functions, and can alleviate the gradient vanishing and explosion problems in the back propagation process, and accelerate the convergence speed of the neural network model. GCN itself may be limited in modeling some complex nonlinear relationships, and the introduction of the maxout structure can improve the model's feature extraction and expression of nonlinear features, and can further improve the image classification accuracy of the neural network model. Therefore, the maxout activation function is combined with GCN to design the maxout-GCN model.
[0108] The present invention designs a dual-channel GCN model with two layers of maxout activation function, inputs the amplitude and phase extracted by Gabor wavelet transform into their respective channels, and extracts more detailed features through multi-layer convolution, pooling layer and maxout activation function. Its structure is as follows Fig.10 shown.
[0109] First, the rectified linear unit (ReLU) function is selected. The ReLU function is an unsaturated nonlinear function that is used to solve problems such as gradient explosion and gradient disappearance, and can also speed up convergence. Its calculation formula is:
[0110] f(x) = max(0, x)
[0111] The design of the present invention is mainly composed of two graph convolution layers, two pooling layers, a ReLU activation function and two maxout function activation layers. The combination of GCN and maxout activation function can treat complex feature patterns more flexibly, and the maxout structure can extract local features and patterns of facial expressions; GCN captures the structural information of facial expressions by learning node representation, and the nonlinear nature of maxout introduces some additional diversity, so that the node representation is richer and adapts to nodes in different types of facial expressions. At the same time, the key frames of the facial expression sequence frames are extracted by combining the GRAY-YCrCb method, and then the amplitude and phase features of the key frames are extracted by Gabor wavelet transform. Finally, the features are extracted by GCN. Among them, the graph convolution layer in GCN is the core part, which is mainly responsible for node representation, and the feature representation of the node is updated by aggregating the neighbor information of each node, which is shown in the following formula.
[0112]
[0113] h i represents the original feature representation of node i; A represents the adjacency matrix with self-connection; d i is the degree of node i; W is the weight matrix, and σ is the activation function.
[0114] This design adopts a dual-channel GCN, in which one channel processes the structural information of the node (adjacency matrix) and the other channel processes the attribute information of the node (node feature vector), so that the model can more comprehensively utilize multi-source information to process heterogeneous features in facial expressions, thereby improving the robustness of the model.
[0115] Finally, softmax is used in the fully connected layer. Softmax is a multi-output competitive classifier that maps some inputs to real numbers between 0 and 1, and normalizes them to ensure that the sum is 1, so the sum of the probabilities of multiple classifications is exactly 1. It is usually used as the output layer activation function for easy calculation.
[0116] The number of neurons in this layer is equal to the number of categories in the sample, and each neuron has a value between 0 and 1. This value represents the probability that the sample belongs to the corresponding category calculated by the neural network. Assume that the amplitude feature corresponding to the i-th expression in the softmax layer is Ai, and the phase feature is Bi. α and β are the weights of the amplitude feature and the phase feature, respectively. If α+β=1, the result can be described as:
[0117]
[0118] Therefore, the combination of dual-channel GCN and maxout activation function can perform feature extraction at two levels: on the one hand, graph structure information and facial features are learned through GCN, and on the other hand, higher-order nonlinear features are introduced through maxout activation function. Especially in complex scenes, the method of using graph structure and image features helps to improve the performance of expression recognition.
[0119] The dual-channel GCN can be used to learn the extracted features, inputting the amplitude and phase features into their respective channels for convolution operations. The structural information channel captures the local structure of the input image; the attribute information channel can use feature vectors to process and fuse information such as facial appearance, color, and texture, so as to effectively extract facial expression information from facial features.
[0120] In order to facilitate understanding of the above embodiments of the present invention, see Fig.11 , Fig.11 The present invention is a flowchart of a method for facial expression recognition according to a specific embodiment of the present invention, wherein the method comprises the following steps:
[0121] Step 1. Get the CK+ facial expression dataset. The CK+ dataset is a facial expression analysis dataset widely used in the fields of computer vision and facial emotion recognition. Its facial expression images are saved in the form of sequence frames. The images in the dataset are high-resolution, usually 640x490 pixels, and in grayscale. It contains 7 expressions: anger, contempt, disgust, fear, happy, sadness, and surprise.
[0122] Step 2: preprocess the acquired facial expression data, including grayscale conversion, face positioning, and cropping.
[0123] Step 3: Extract key frame images. The pre-processed images are subjected to median filtering, Gaussian filtering and the Gray-YCrCb method (see the description in the above embodiment for details) to extract the last three key frames of the five frames in the sequence, which can be used as the subsequent data set.
[0124] The extracted key frames are input into the constructed DNN model, such as Fig.12As shown in the figure, it includes an input layer, three hidden layers and an output layer. The 64*64 facial expression image is put into the DNN model for experiment, and the optimization strategy uses the Stochastic Gradient Descent (SGD) optimization algorithm. The idea of SGD is to randomly extract a sample to update the parameters, which greatly improves the update speed of each round of parameters and reduces the computational overhead. Put the 224*224 facial expression image into the VGG16 model, as shown in the figure. Fig.13 As shown in Figure 2, VGG16 convolution cascade has fewer parameters than using a larger convolution kernel alone, and has more nonlinear transformations than a single convolution layer.
[0125] Through the above experiments, the effectiveness of key frames in DNN and VGG16 is verified, which can enhance the feature information of facial expressions, filter the unique information of the face itself, such as wrinkles, and further improve the accuracy of facial expression recognition.
[0126] Step 4: Perform Gabor wavelet transform on the key frame image to obtain amplitude and phase characteristics.
[0127] Step 5: Input the extracted amplitude and phase features into maxout-GCN for training. Finally, a softmax multi-classifier is used in the top-level classifier to classify and realize facial expression recognition, so as to further improve the accuracy of facial expression recognition.
[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0129] In this embodiment, a facial expression recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0130] Fig.14 is a structural diagram of a facial expression recognition device according to an embodiment of the present invention, such as Fig.14 As shown, the device comprises:
[0131] The first acquisition module 1401 is used to acquire a video image containing a face image;
[0132] A second acquisition module 1402 is used to acquire a target frame image in the video image, wherein the feature information of the target frame image is more than the feature information of other frame images, and the other frame images are images other than the target frame image in the video image;
[0133] An extraction module 1403 is used to extract features from the target frame image to obtain amplitude features and phase features;
[0134] The result output module 1404 is used to input the amplitude feature and the phase feature into a neural network to obtain an expression recognition result.
[0135] Optionally, the second acquisition module includes:
[0136] A first acquisition submodule, used for acquiring a first image in which the facial expression of the human face is a neutral expression in the video image;
[0137] A determination submodule, used to determine the grayscale value of the first image;
[0138] The second acquisition submodule is used to acquire, in the video image, the target frame image whose grayscale value difference with the grayscale value of the first image is within a first preset range.
[0139] Optionally, the device further comprises:
[0140] A third acquisition module is used to acquire a grayscale value of a second image in the video image, where the second image is an image in the video image other than the first image, and the second image includes at least one frame of image;
[0141] A conversion module, used for converting the grayscale value of the first image and the grayscale value of the second image into color space values respectively, so as to obtain the first image and the second image containing the facial contour features of the face image;
[0142] The second acquisition submodule includes:
[0143] An acquisition unit, configured to acquire a target difference value in which the difference between the second grayscale value and the first grayscale value is within the first preset range;
[0144] A determination unit is used to determine the target frame image corresponding to the target difference value.
[0145] Optionally, the video image is a sequence frame image of the face image, and the first image is a first frame image in the sequence frame image; and the acquisition unit includes:
[0146] A first acquisition subunit, configured to sequentially acquire at least two differences between the second grayscale value and the first grayscale value according to the order in which the sequence frame images are arranged;
[0147] The second acquisition subunit is used to acquire a target difference value arranged in a second preset range from the at least two difference values.
[0148] Optionally, the result output module is specifically used for:
[0149] Inputting the amplitude feature and the phase feature into a dual-channel neural network model to obtain an expression recognition result;
[0150] Among them, one channel of the dual-channel neural network model is used to obtain local features of the face image, and the other channel of the dual-channel neural network model is used to fuse the acquired facial features of the face image to obtain facial expression features.
[0151] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0152] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0153] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0154] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0155] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0156] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0157] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for recognizing facial expressions, characterized in that: include: Acquire a video image containing a face image; Acquire a target frame image in the video image, wherein the target frame image has more feature information than other frame images, the other frame images are images in the video image other than the target frame image, and the target frame image is determined based on the difference in grayscale values between frame images; Feature extraction is performed on the target frame image to obtain amplitude features and phase features, and key frames of facial expression sequence frames are extracted by a GRAY-YCrCb method, wherein the GRAY-YCrCb method includes: obtaining a grayscale image of the target frame image, and converting the grayscale value of the grayscale image into a BGR color space value; converting the target frame image into a BGR image based on the BGR color space value and a first formula; obtaining the grayscale value of the BGR image based on a second formula; the first formula includes: ; The second formula includes: ; Inputting the amplitude feature, the phase feature and the key frame of the facial expression sequence frame into a neural network to obtain an expression recognition result; The step of inputting the amplitude feature and the phase feature into a neural network to obtain an expression recognition result includes: Inputting the amplitude feature and the phase feature into a dual-channel neural network model to obtain an expression recognition result; Among them, one channel of the dual-channel neural network model is used to process the pixel points of the target frame image as the structural information of the image nodes, and the other channel of the dual-channel neural network model is used to process the node feature vector of the image node; and one channel of the dual-channel neural network model is used to obtain the local features of the face image, and the other channel of the dual-channel neural network model is used to fuse the acquired facial features of the face image to obtain facial expression features; the dual-channel neural network model includes two layers of maxout activation functions and two GCN graph convolution layers, and both channels are configured with the maxout activation function.
2. The method according to claim 1, characterized in that The step of acquiring a target frame image from the video image comprises: Acquire a first image in which the facial expression of the human face is a neutral expression in the video image; determining a grayscale value of the first image; The target frame image is obtained in the video image, the difference between the grayscale value of the target frame image and the grayscale value of the first image being within a first preset range.
3. The method according to claim 2, characterized in that Before obtaining the target frame image in the video image, the difference between the grayscale value and the grayscale value of the first image being within a first preset range, the method further includes: Acquire a grayscale value of a second image in the video image, where the second image is an image in the video image other than the first image, and the second image includes at least one frame of image; Converting the grayscale value of the first image and the grayscale value of the second image into color space values respectively, to obtain the first image and the second image containing facial contour features of the face image; Acquire a first grayscale value based on the first image including the facial contour features of the facial image, and acquire a second grayscale value based on the second image including the facial contour features of the facial image; The step of obtaining the target frame image in the video image, the target frame image having a grayscale value whose difference with the grayscale value of the first image is within a first preset range, comprises: Acquire a target difference value where the difference between the second grayscale value and the first grayscale value is within the first preset range; Determine the target frame image corresponding to the target difference value.
4. The method according to claim 3, characterized in that The video image is a sequence frame image of the face image, and the first image is a first frame image in the sequence frame image; and obtaining a target difference value where the difference between the second grayscale value and the first grayscale value is within the first preset range includes: According to the order in which the sequence frame images are arranged, sequentially acquiring at least two differences between the second grayscale value and the first grayscale value; A target difference value arranged within a second preset range is obtained from the at least two difference values.
5. A facial expression recognition device, characterized in that: include: A first acquisition module, used to acquire a video image containing a face image; A second acquisition module is used to acquire a target frame image from the video image, wherein the feature information of the target frame image is more than the feature information of other frame images, the other frame images are images in the video image other than the target frame image, and the target frame image is determined based on the difference in grayscale values between frame images; The extraction module is used to extract features of the target frame image to obtain amplitude features and phase features, and extract key frames of facial expression sequence frames by using the GRAY-YCrCb method, wherein the GRAY-YCrCb method includes: obtaining a grayscale image of the target frame image, and converting the grayscale value of the grayscale image into a BGR color space value; converting the target frame image into a BGR image based on the BGR color space value and a first formula; obtaining the grayscale value of the BGR image based on a second formula; the first formula includes: ; The second formula includes: ; A result output module is used to input the amplitude feature, the phase feature and the key frame of the facial expression sequence frame into a neural network to obtain an expression recognition result; The step of inputting the amplitude feature and the phase feature into a neural network to obtain an expression recognition result includes: Inputting the amplitude feature and the phase feature into a dual-channel neural network model to obtain an expression recognition result; Among them, one channel of the dual-channel neural network model is used to process the pixel points of the target frame image as the structural information of the image nodes, and the other channel of the dual-channel neural network model is used to process the node feature vector of the image node; and one channel of the dual-channel neural network model is used to obtain the local features of the face image, and the other channel of the dual-channel neural network model is used to fuse the acquired facial features of the face image to obtain facial expression features; the dual-channel neural network model includes two layers of maxout activation functions and two GCN graph convolution layers, and both channels are configured with the maxout activation function.
6. The device according to claim 5, characterized in that The second acquisition module includes: A first acquisition submodule, used for acquiring a first image in which the facial expression of the human face is a neutral expression in the video image; A determination submodule, used to determine the grayscale value of the first image; The second acquisition submodule is used to acquire, in the video image, the target frame image whose grayscale value difference with the grayscale value of the first image is within a first preset range.
7. The device according to claim 6, characterized in that The device also includes: A third acquisition module is used to acquire the gray value of a second image in the video image before acquiring the target frame image whose gray value difference with the gray value of the first image is within a first preset range in the video image, wherein the second image is an image in the video image other than the first image, and the second image includes at least one frame of image; A conversion module, used for converting the grayscale value of the first image and the grayscale value of the second image into color space values respectively, so as to obtain the first image and the second image containing the facial contour features of the face image; A grayscale value determination module, configured to obtain a first grayscale value based on the first image including the facial contour features of the facial image, and to obtain a second grayscale value based on the second image including the facial contour features of the facial image; The second acquisition submodule includes: An acquisition unit, configured to acquire a target difference value in which the difference between the second grayscale value and the first grayscale value is within the first preset range; A determination unit is used to determine the target frame image corresponding to the target difference value.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 4 when executed.
9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.
