A facial expression recognition method based on quaternion Zernike moment invariants

Abstract

The present invention discloses a facial expression recognition method based on quaternion Zernike moment invariants. The method comprises the following steps: Step 1: Acquire a color facial expression image to be verified; Step 2: Divide the image into nine small blocks of equal size; Step 3: Extract 0-14 order quaternion Zernike moment invariant features for each small block; Step 4: Design a neural network based on multi-local representation and Transformer; Step 5: Input the recognition matrix into a deep neural network for training and recognition. The present invention uses the color facial features extracted by quaternion Zernike moments to have high robustness. By utilizing machine vision and deep learning technology, it effectively realizes the recognition of seven facial expressions, including anger, disgust, and happiness.

Description

Technical Field

[0001] The invention relates to a facial expression recognition method based on quaternion Zernike moment invariants, and belongs to the technical field of image processing. Background Art

[0002] Facial expressions are the most common, direct, and natural way to express human psychology and emotions in non-verbal communication. Consequently, they are widely used in numerous human-computer interaction scenarios, such as intelligent robots, virtual reality, industry analysis, and traffic safety. However, just as human psychological states are diverse and complex, facial expressions are also diverse. In addition to the seven common expressions—happiness, surprise, sadness, anger, disgust, fear, and neutrality—there are also a variety of distinguishable compound expressions, such as surprise (happiness + surprise) and grief and indignation (sadness + anger). Therefore, the current accuracy of facial expression recognition is low, and improving this accuracy is a technical problem that those skilled in the art urgently need to address. Summary of the Invention

[0003] Technical problem: The purpose of the present invention is to provide a facial expression recognition method based on quaternion Zernike moment invariants, which can improve the accuracy of expression recognition.

[0004] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0005] A facial expression recognition method based on quaternion Zernike moment invariants comprises the following steps:

[0006] Step 1: Collect the color facial expression image to be verified;

[0007] Step 2: Divide the image into 9 small blocks of equal size;

[0008] Step 3: Extract the 0-14th order quaternion Zernike moment invariant features for each small block;

[0009] Step 3.1: Extract the 0-14th order Zernike moment features for the R, G, and B channels of each 0-block;

[0010] Step 3.2: Use a fast calculation method to obtain the moment features of the three channels, and then obtain the quaternion Zernike moment features;

[0011] Step 3.3: Extract the rotation invariant features of the quaternion Zernike moment features of the 9 small blocks and then cascade them, that is, the recognition matrix of the color facial expression image;

[0012] Step 4: Design of neural network based on multiple local representations and Transformer;

[0013] Step 5: Input the recognition matrix into the deep neural network for training and recognition.

[0014] Furthermore, the algorithm for constructing the Zernike moments described in step 3.1 is as follows:

[0015] For a two-dimensional grayscale image f(r,θ) in polar coordinates, its order is n and its repeatability is m, the Zernike moment is:

[0016]

[0017] Where V n,m (r,θ) is the Zernike basis function, defined as follows:

[0018] V n，m (r, θ) = R n，m (r)e -imθ , |r|≤1.

[0019] i is an imaginary number, R n，m (r) is a real-valued radial polynomial expressed as follows:

[0020]

[0021] Where n is a non-negative integer, m is an integer, satisfying |m|≤n, and n-|m| is an even number;

[0022] For an N×N grayscale discrete image, its Zernike moment is defined as:

[0023]

[0024] in:

[0025]

[0026]

[0027] Furthermore, the quaternion Zernike moment in step 3 is defined as:

[0028] definition is the right quaternion Zernike moment of the color image with order n and repeatability m, then:

[0029]

[0030] Where |m|≤n, and n-|m| is an even number, μ is the unit quaternion, that is, μ=ai+bj+ck, where i, j and k are traditional imaginary symbols, and ||μ||=1.

[0031] Similarly, we can define is the left quaternion Zernike moment of the color image of order n and repeatability m:

[0032]

[0033] Furthermore, in step 3.2, the method for quickly calculating the quaternion Zernike moment is derived from the relationship between the Zernike moment and the quaternion Zernike moment. The specific calculation formula is as follows:

[0034] Define f R (r,θ),f G (r,θ),f B (r, θ) are the pixel values ​​of the image in the three channels of RGB, Z n，m (f R ) ， Z n，m (f G ), Z n，m (f B ) are the Zernike moments of order n and repeatability m on each channel, μ is the unit quaternion, and we take Then we have:

[0035]

[0036] in:

[0037]

[0038]

[0039]

[0040]

[0041] Furthermore, in step 3.3, the invariant features of the moment are extracted from the moment feature vector, and the method for constructing the moment is as follows:

[0042] Define the original image as f, and the image after rotation by any angle α as f′, then f′(r,θ)=f(r,θ-α), so we have:

[0043]

[0044] make Then we have:

[0045]

[0046] therefore This is the rotation invariant of the quaternion Zernike moment we constructed.

[0047] Furthermore, the recognition matrix of the color facial expression image described in step 3.3 is extracted as follows:

[0048] The color facial expression image to be recognized is cropped and divided into 9 small blocks of equal size. The rotation invariant of the 0-14th order quaternion Zernike moment is extracted from each small block, and the final facial expression image recognition vector is obtained by cascading.

[0049] Furthermore, the neural network design based on multiple local representations and Transformer in step 4 has the following specific network structure:

[0050] The input of the network is the facial expression recognition vector, a matrix of size 24×24×4, which is then input into 8 LANets to obtain 8 attention maps. The attention map is then input into the Random Multi-Attention Dropping (RMAD) algorithm. Then, through the MAX function, multiple attention maps (M1 to M8) are concentrated into one map (M-out). The attention map (M-out) and the original input are then multiplied element-by-element to obtain a new 24×24×4 matrix. The matrix is ​​divided into 9 blocks and passed into the Encoder module of the Transformer. Finally, a fully connected layer and a classification layer are connected.

[0051] The LANet structure is: two 1×1 convolution kernels connected in series, with a Sigmoid activation function added at the end. RMAD works by randomly selecting one channel from an input of N channels and setting all eigenvalues ​​to 0. The MAX function structure is:

[0052] M-out(x,y)=MAX{M p (x,y)},

[0053] The value of p ranges from 1 to 8, and M1 to M8 are all attention maps.

[0054] Compared to existing technologies, the present invention offers the following advantages: The color facial features extracted using quaternion Zernike moments are highly robust. Leveraging machine vision and Transformer-based deep learning technology, the method effectively recognizes seven facial expressions, including anger, disgust, and happiness. Furthermore, experiments using the method on the public KDEF dataset demonstrated an effective recognition rate of 99.08%. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 : Flowchart of the present invention;

[0056] Figure 2 : The structural diagram of the neural network designed in the present invention;

[0057] Figure 3 : The recognition results of this invention on the KDEF public dataset. DETAILED DESCRIPTION

[0058] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only intended to illustrate the present invention and are not intended to limit the scope of the present invention. After reading this invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims appended hereto.

[0059] Example 1: Figure 1 As shown, a facial expression recognition method based on quaternion Zernike moment invariants includes the following steps:

[0060] Step 1: Collect the color facial expression image to be verified;

[0061] Step 2: Divide the image into 9 small blocks of equal size;

[0062] Step 3: Extract the 0-14th order quaternion Zernike moment invariant features for each small block;

[0063] The quaternion Zernike moments in step 3 are defined as:

[0064] definition For the right quaternion Zernike moment of the color image with order n and repeatability m, we have:

[0065]

[0066] Where |m|≤n, and n-|m| is an even number, μ is the unit quaternion, that is, μ=ai+bj+ck, where i, j and k are traditional imaginary symbols, and ||μ||=1.

[0067] Similarly, we can define is the left quaternion Zernike moment of the color image of order n and repeatability m:

[0068]

[0069] At the same time, step 3 includes the following three operations:

[0070] Step 3.1: Extract the 0-14th order Zernike moment features for the R, G, and B channels of each small block;

[0071] The construction method of the moment transform in step 3.1 is as follows:

[0072] For a two-dimensional grayscale image f(r,θ) in polar coordinates, its order is n and its repeatability is m, the Zernike moment is:

[0073]

[0074] Where V n,m (r,θ) is the Zernike basis function, defined as follows:

[0075] V n,m (r,θ)=R n,m (r)e -imθ ,|r|≤1.

[0076] i is an imaginary number, R n,m (r) is a real-valued radial polynomial expressed as follows:

[0077]

[0078] Where n is a non-negative integer, m is an integer, satisfying |m|≤n, and n-|m| is an even number;

[0079] For an N×N grayscale discrete image, its Zernike moment is defined as:

[0080]

[0081] in:

[0082]

[0083]

[0084] Step 3.2: Use a fast calculation method to obtain the moment features of the three channels, and then obtain the quaternion Zernike moment features;

[0085] The fast calculation method of the moment in step 3.2 is derived from the relationship between the Zernike moment and the quaternion Zernike moment. The specific calculation formula is as follows:

[0086] Define f R (r,θ),f G (r,θ),f B (r, θ) are the pixel values ​​of the image in the three channels of RGB, z n，m (f R ), Z n，m (f G ), Z n，m (f B ) are the Zernike moments of order n and repeatability m on each channel, μ is the unit quaternion, and we take Then we have:

[0087]

[0088]

[0089] in:

[0090]

[0091]

[0092]

[0093]

[0094] Step 3.3: Extract the rotation invariant features of the quaternion Zernike moment features of the 9 small blocks and then cascade them, that is, the recognition matrix of the color facial expression image;

[0095] The specific operation of extracting the quaternion moment invariant features in step 3.3 is as follows:

[0096] Define the original image as f, and the image after rotation by any angle α as f′, then f′(r,θ)=f(r,θ-α), so we have:

[0097]

[0098]

[0099] make Then we have:

[0100]

[0101] therefore This is the rotation invariant of the quaternion Zernike moment we constructed.

[0102] Step 4: Design of neural network based on multiple local representations and Transformer;

[0103] The specific network structure in step 4 is as follows Figure 2 As shown:

[0104] The input of the network is the facial expression recognition vector, a matrix of size 24×24×4, which is then input into 8 LANets to obtain 8 attention maps. The attention map is then input into the Random Multi-Attention Dropping (RMAD) algorithm. Then, through the MAX function, multiple attention maps (M1 to M8) are concentrated into one map (M-out). The attention map (M-out) and the original input are then multiplied element-by-element to obtain a new 24×24×4 matrix. The matrix is ​​divided into 9 blocks and passed into the Encoder module of the Transformer. Finally, a fully connected layer and a classification layer are connected.

[0105] The LANet structure is: two 1×1 convolution kernels connected in series, with a Sigmoid activation function added at the end. RMAD works by randomly selecting one channel from an input of N channels and setting all eigenvalues ​​to 0. The MAX function structure is:

[0106] M-out(x,y)=MAX{M p (x,y)},

[0107] The value of p ranges from 1 to 8, and M1 to M8 are all attention maps.

[0108] Step 5: Input the recognition matrix into the deep neural network for training and recognition.

[0109] Example 2: The steps of the present invention are described in detail below using the KDEF dataset as an example.

[0110] 1. Experimental conditions:

[0111] The verification experiments were conducted on a 64-bit computer with an Intel(R) Core(TM) i7-4790CPU@3.60GHz×8 processor, a GeForce GT705 / PCIe / SSE2 graphics card, Ubuntu 16.04LTS operating system, Python 3.6 programming language, and PyTorch (1.11.0) runtime environment.

[0112] 2. Experimental subjects:

[0113] The KDEF dataset consists of 1960 color images of facial expressions from 7 classes. Each image represents only one expression category, and these classes are completely mutually exclusive.

[0114] 3. Dataset preprocessing:

[0115] The dataset is randomly divided into training and test sets in a ratio of 8:2. Each image is rotated and the dataset size is expanded to 8820 images. Each image is then cut into 9 blocks. The values ​​of the 0-14th order quaternion Zernike moment invariants of each block are then calculated. Finally, the matrices are concatenated to form the facial expression recognition matrix of the image, which is 24×24×4 in size. These matrices will become the input features of the subsequent network.

[0116] 4. Experimental methods:

[0117] The recognition matrix, with a size of 24×24×4, is first input into the LANet, passed through two 1×1 convolution kernels, and then through a sigmoid activation function, resulting in eight attention maps. These attention maps are then fed into the Random Multi-Attention Dropout (RMAD) module. This encourages the model to explore comprehensive local patches in addition to the most discriminative ones, adaptively focusing on different local patches. The MAX module then aggregates the multiple attention maps to generate a single attention map. Specifically, an element-wise maximum operation is used to aggregate the multiple attention maps. Finally, M-out is element-wise multiplied by the original feature matrix. This suppresses unimportant regions in the original feature map, and vice versa.

[0118] Then, M-out is input into the Reshape module to obtain 9 8×8×4 blocks, representing the 9 blocks of the original input. The blocks are added with position information (the block in the upper left corner has position information (0, 0), and so on) and input into the TranformerEncoder module. In this experiment, the Encoder module is repeated 6 times, and finally an MLP Head layer and a classification layer are connected to obtain the final output.

[0119] Neural network model parameter setting process: During training, Adam optimization was used, with an initial learning rate of 5e-5, a batch size of 64, Adam's beta1 parameter set to 0.9, beta2 parameter set to 0.999, epsilon set to 1e-8, and a learning rate decay rate set to 0.001. Training was performed for a total of 500 epochs.

[0120] 5. Experimental results:

[0121] For image classification tasks, the average accuracy of all predictions is calculated. The higher the average accuracy, the higher the classification accuracy of the model, and the better the model effect. The experimental results are as follows: Figure 3 shown.

[0122] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on quaternion Zernike moment invariants, characterized in that: The following steps are involved: Step 1: Collect the color facial expression image to be verified; Step 2: Divide the image into 9 small blocks of equal size; Step 3: Extract the 0-14th order quaternion Zernike moment invariant features for each small block; Step 3.1: Extract the 0-14th order Zernike moment features for the R, G, and B channels of each small block; Step 3.2: Use a fast calculation method to obtain the moment features of the three channels, and then obtain the quaternion Zernike moment features; Step 3.3: Extract the rotation invariant features of the quaternion Zernike moment features of the 9 small blocks and then cascade them, that is, the recognition matrix of the color facial expression image; Step 4: Design of neural network based on multiple local representations and Transformer; Step 5: Input the recognition matrix into the deep neural network for training and recognition; Among them, in step 4, based on the multi-local representation and Transformer neural network design, the network structure is: The input of the network is the facial expression recognition vector, a matrix of size 24×24×4, which is then input into 8 LANets to obtain 8 attention maps. The attention map is then input into the Random Multi-Attention Dropping algorithm, namely RMAD. Then, through the MAX function, multiple attention maps M1 to M8 are concentrated into a single map M-out. The attention map M-out is then multiplied element-by-element by the original input to obtain a new 24×24×4 matrix. The matrix is ​​divided into 9 blocks and passed to the Encoder module of the Transformer. Finally, a fully connected layer and a classification layer are connected. The working principle of RMAD is: for N channels of input, randomly select one of the channels and set all eigenvalues ​​to 0. The structure of the MAX function is: M-out(x,y)=MAX{M p (x,y)}, The value of p ranges from 1 to 8, and M1 to M8 are all attention maps; In step 4, the structure of LANet is: two 1×1 convolution kernels are connected in series, and a Sigmoid activation function is added at the end.

2. The facial expression recognition method based on quaternion Zernike moment invariants according to claim 1, wherein The algorithm for constructing the Zernike moments in step 3.1 is as follows: For a two-dimensional grayscale image f(r,θ) in polar coordinates, its order is n and its repeatability is m, the Zernike moment is: Where V n,m (r,θ) is the Zernike basis function, defined as follows: V n,m (r,θ))R n,m (bid -imθ ,|r|≤1, i is an imaginary number, R n,m (r) is a real-valued radial polynomial expressed as follows: Where n is a non-negative integer, m is an integer, satisfying |m|≤n, and n-|m| is an even number; For an N×N grayscale discrete image, its Zernike moment is defined as: in:

3. The facial expression recognition method based on quaternion Zernike moment invariants according to claim 1, wherein The quaternion Zernike moments in step 3 are defined as: definition For the right quaternion Zernike moment of the color image with order n and repeatability m, we have: Where |m|≤n, and n-|m| is an even number, μ is the unit quaternion, that is, μ=ai+bj+ck, where i, j and k are traditional imaginary symbols, and ||μ||=1; definition is the left quaternion Zernike moment of the color image of order n and repeatability m:

4. The method for facial expression recognition based on quaternion Zernike moment invariants according to claim 1, wherein The method for quickly calculating the quaternion Zernike moment in step 3.2 is derived from the relationship between the Zernike moment and the quaternion Zernike moment. The specific calculation formula is as follows: Define f R (r,θ),f G (r,θ),f B (r, θ) are the pixel values ​​of the image in the three channels of RGB, Z n,m (f R ),Z n,m (f G ),Z n,m (f B ) are the Zernike moments of nth order and mth degree of repetition on each channel, μ is the unit quaternion, and Then we have: in:

5. The facial expression recognition method based on quaternion Zernike moment invariants according to claim 1, wherein In step 3.3, the invariant features of the moment are extracted from the moment eigenvector, and the method for constructing the moment invariant is as follows: Define the original image as f, and the image after rotation by any angle α as f′, then f′(r,θ)=f(r,θ-α), so: make Then we have: That is the rotation invariant of the constructed quaternion Zernike moment.

6. The facial expression recognition method based on quaternion Zernike moment invariants according to claim 1, wherein The recognition matrix of the color facial expression image in step 3.3 is extracted as follows: the color facial expression image to be recognized is cropped and divided into 9 small blocks of equal size. The rotation invariants of the 0-14th order quaternion Zernike moments are extracted for each small block, and the final facial expression image recognition vector is obtained by cascading.

Images