Facial expression recognition method based on two-dimensional multi-manifold discriminant analysis algorithm

Abstract

The invention discloses a facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm, and belongs to the field of computer vision and pattern recognition. The method comprises the following steps: firstly, extracting a face salient region by using an active shape model, adjusting the size of the face salient region, and performing frequency domain conversion through two-dimensional fractional Fourier transform; then, a local relation matrix in the manifolds and a local relation matrix between the manifolds are constructed, and an objective function is defined to reflect change information in the manifolds, similarity information in the manifolds and edge information between the manifolds; based on a difference criterion optimization mode and an entropy criterion optimization mode, calculation is carried out to obtain an identification matrix; compared with a traditional method, the method has the advantages that the saliency of the face region is fully considered, and the robustness and generalization ability of recognition are enhanced by weighing the similarity among the same kind of samples, the edge information among the different kinds of samples and the change information among the same kind of samples. Experimental results show that the method has significant advantages in the field of expression recognition.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and pattern recognition, and in particular relates to a facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm. Background Art

[0002] Currently, widely used expression recognition methods primarily extract discriminant features from two-dimensional image data. This approach helps avoid the "small sample size" problem encountered in one-dimensional methods during feature extraction. Yang et al., who first proposed the two-dimensional principal component analysis (2DPCA) algorithm, applied it to facial feature extraction and claimed that 2DPCA could easily and directly extract features from images. Experiments have demonstrated that 2DPCA is superior to traditional PCA in recognition results. Subsequently, researchers proposed a two-dimensional feature extraction and classification method based on linear discriminant analysis (LDA). For example, Li et al. used the Fisher discriminant analysis method to extract features directly from the image matrix; the Uncorrelated Image Matrix Linear Discriminant Analysis (IMLDA) algorithm was proposed as a special form of 2DLDA, which can eliminate the correlation between discriminant vectors and has better recognition performance than LDA in the case of small samples; Yang et al. proposed a bidirectional (rows and columns of the image) two-dimensional discriminant feature extraction method based on IMLDA, which can compress the discrimination information to the upper left part of the sample matrix; Ye et al. used an iterative method to simultaneously extract discrimination information in two directions, etc.

[0003] In order to increase the richness of identification information, researchers have also proposed more two-dimensional algorithms based on manifold learning, which are designed for feature extraction and classification, such as Two-Dimensional Locality Preserving Projection (2D-LPP), Two-Dimensional Discriminant Locality Preserving Projections (2DDLPP), Two-Dimensional Discriminations and Supervision Locality Preserving Projection (2DDSLPP), Two-Dimensional Local Graph Embedding Discriminant Analysis (2DLGEDA), Two-Directional Two-Dimensional Discriminant Locality Preserving Projections ((2D)2-DLPP), Two-Dimensional Local Similarity and Diversity Projection (2DSLSDP), Two-Dimensional Nearest Preserving Projections, 2DNPP), Two-Dimensional Neighborhood Margin and Variation Embedding (2DNMVE) and other algorithms.

[0004] Most facial expression recognition methods typically convert the original two-dimensional facial image into one-dimensional data (vectors) for processing. However, this conversion significantly increases the image dimensionality, requiring more discriminant information for better recognition. This not only increases the training and recognition burden but also places a significant strain on the overall algorithm's computational overhead. Furthermore, the "straightening" operation destroys the structural information in the two-dimensional image, which can be crucial for facial expression recognition.

[0005] Most algorithms reduce the training set data to a single manifold space, extracting only a discriminant matrix for classification and recognition. However, it remains unknown whether manifolds can effectively represent high-dimensional data structures. To address this issue, Xiao et al. proposed a multi-manifold learning method for expression recognition (Facial Expression Recognition Based on Multiple Manifolds [J], Pattern Recognition, 2011, Vol. 44, No. 1: pp. 107-116). They claim that different expressions may reside in different manifolds, so each manifold can be "learned" separately. Experimental verification shows that the multi-manifold method achieves better classification performance than the single-manifold method. Meanwhile, Lu et al. proposed a Discriminative Multi-Manifold Analysis (DMMA) method to address the single-person, single-sample facial recognition problem. DMMA evenly divides each facial image into local blocks to form the training set and extracts the discriminant matrix for each person. Their experiments demonstrate the effectiveness of this algorithm.

[0006] At the same time, most methods minimize the similarity between samples of the same class by minimizing the distance between samples within a class. However, this approach may result in the loss of variation information between samples of the same class, which in turn makes the distance between samples of the same class even smaller in low-dimensional space, potentially leading to the "embedding curse" and "overfitting" problems. Therefore, unrestricted "compression" of variation information between samples of the same class may not achieve the ultimate goal of improving the algorithm's recognition performance. To address this issue, Gao et al. proposed the Enhanced Fisher Discriminant Criterion (EFDC). This method combines intra- and inter-class variation information and uses the Fisher Discriminant Criterion to reduce the dimensionality of the original sample space. Subsequently, Gao et al. proposed a two-dimensional edge information embedding method to preserve variation information between the same expressions. Beat et al. combined three types of local information (local similarity information, local intra-class variation information, and local inter-class difference information) to propose a feature extraction and classification algorithm. Although Gao et al.'s method utilizes global information of training samples and local information of samples of the same class, it cannot guarantee that the distance between samples in the low-dimensional space meets the classification requirements. Furthermore, the method proposed by Gao et al. cannot guarantee that a single manifold can fully describe the local structural information of the entire training set. This is because the dimensionality reduction achieved by the above method is smaller than that achieved by the one-dimensional method, resulting in excessive "compression" of the high-dimensional space and loss of geometric information of the high-dimensional data.

[0007] Therefore, it is of great significance to carry out research on facial expression recognition based on two-dimensional multi-manifold discriminant analysis algorithm, so as to overcome the interference of factors such as facial posture and lighting on expression recognition, better capture the subtle changes in expression information, reasonably mine and utilize information within and between manifolds, avoid problems such as "overfitting", and thus improve the overall performance of expression recognition. Summary of the Invention

[0008] The present invention aims to provide a facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm, aiming to improve the accuracy and robustness of facial expression recognition, especially in the case of processing multi-category samples and facial regions with different salience, with higher recognition performance.

[0009] To achieve the above object, the technical solution adopted in the present invention is: a facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm, comprising the following steps:

[0010] Step S1: extract multiple salient regions of the facial image using the active shape model, including the left eye, right eye, left cheek, right cheek, and mouth, classify them into an expression database, and adjust the salient regions of each sample to a uniform size to form a set of salient regions of the same size;

[0011] Step S2: using a two-dimensional fractional Fourier transform to perform frequency domain conversion on each salient region so that the image is jointly expressed in the time-frequency plane;

[0012] Step S3: construct the set of salient regions of each expression category into an independent manifold to form a multi-manifold model, whose overall dimension is the product of the dimension of a single salient region and the number of salient regions;

[0013] Step S4, designing an objective function, including a manifold variation information maximization item, a manifold similarity information minimization item, and an inter-manifold edge information maximization item, wherein: the manifold variation information maximization item is calculated by weighting the local neighbor differences of similar samples; the manifold similarity information minimization item is calculated by weighting the local neighbor similarities of similar samples; and the inter-manifold edge information maximization item is calculated by weighting the local neighbor differences of heterogeneous samples;

[0014] Step S5: optimizing the objective function based on the difference criterion or the entropy criterion, and calculating the discrimination matrix corresponding to each type of expression;

[0015] Step S6: Project the salient area of ​​the test sample into a low-dimensional space, and use the discriminant matrix to perform facial expression classification.

[0016] Furthermore, the order parameters P1 and P2 of the two-dimensional fractional Fourier transform are both set to 0.5.

[0017] Furthermore, the specific implementation process of step S2 is:

[0018] Step S21: perform one-dimensional discrete fractional Fourier transform according to the following formula:

[0019] Where, is an imaginary unit; Indicates the angle of rotation in the time-frequency plane; p is the transformation order of FrFT, and p≠2n, n is an integer; F p represents the p-order fractional Fourier transform operator; f p(u) K represents the function of variable u obtained by processing the function f(u′) by p-order fractional Fourier transform; p (u,u′) is the corresponding kernel function;

[0020] Step S22: perform a two-dimensional fractional Fourier transform using the following discrete form:

[0021] Where x(p,q) is the original image, p and q are the row and column indices of the original image respectively; the image size is M×N; is the output image obtained after two-dimensional fractional Fourier transform, where m and n are the row and column coordinates of the output image respectively; is the two-dimensional fractional Fourier kernel function.

[0022] Furthermore, the mathematical expression of the objective function in step S4 is:

[0023] Where W k is the discrimination matrix corresponding to the k-th expression, max J d (W k ) represents the maximization of the change information within the manifold, min J w (W k ) represents the minimization of similarity information within the manifold, max J b (W k ) represents the maximization of edge information between manifolds, is the vth significant region feature vector in the kth expression manifold, in formula (a) express The closest k in the same expression manifold d Nearest neighbor samples, in formula (b) express k in the same expression manifold w similar neighbor samples, in formula (c) express The nearest k in heterogeneous expression manifolds b Nearest neighbor samples, express and The change weight between express and The similarity weight between express and The boundary clarity weight between For K with similar expressions d Neighbor set, For K with similar expressions w Neighbor set, For K with different expressions b The nearest neighbor set, k d 、k w 、k b is the corresponding number of selected neighbors.

[0024] Furthermore, the optimization objective function of the difference criterion in step S5 is:

[0025] Where λ k is the balancing factor, and its value range is [0,1]. c is the number of expression categories in the expression database, and k = 1, 2,…, c.

[0026] Furthermore, the optimization objective function of the entropy criterion in step S5 is:

[0027] Where λ k is the balancing factor, and its value range is [0,1]. c is the number of expression categories in the expression database, and k = 1, 2,…, c.

[0028] Furthermore, the balance factor λ k Calculated by matrix trace ratio:

[0029] Where, and are the similarity matrix and change matrix within the manifold, is the edge matrix between manifolds, and trace(·) represents the trace of the matrix.

[0030] Furthermore, in step S5, the discrimination matrix W is solved by the eigenvalue decomposition method. k, select the eigenvectors that satisfy the following conditions:

[0031] Where, and are the similarity matrix and change matrix within the manifold, is the edge matrix between manifolds; trace(·) represents the trace of the matrix; is the i-th projection direction of the k-th expression, λ is the balance factor, and a is the height of the salient area.

[0032] Furthermore, and The calculation formula is as follows:

[0033] Where κ is the scale parameter, ||·|| is the Euclidean norm, is the vth significant region feature vector in the kth expression manifold, is the rth local neighbor sample with the largest difference in the same expression manifold, is the rth local neighbor sample with similar structure in the same expression manifold, is the rth local neighbor sample close to the boundary in the heterogeneous expression manifold.

[0034] Furthermore, the low-dimensional feature dimension d after projection in step S6 k It is dynamically determined by the trace ratio criterion to ensure that the maximum inter-manifold edge information is preserved.

[0035] The beneficial effects of the above scheme are:

[0036] (1) The present invention focuses on the salient areas of the face, performs time-frequency joint analysis through two-dimensional fractional Fourier transform to enhance local texture and structural information, and constructs local relationship matrices within and between manifolds. It quantifies the relationship between feature vectors through a specific nearest neighbor matrix, overcomes the influence of factors such as facial posture and lighting on expression recognition, can effectively deal with noise and changes in complex environments, extract expression features more accurately, ensure the stability and accuracy of expression recognition, and broaden the application scenarios of expression recognition technology.

[0037] (2) TDM of the present invention 2 The DA algorithm designs a unique objective function, which describes the relationship between expression features from the perspectives of intra-manifold variation information, intra-manifold similarity, and inter-manifold edge information, while minimizing the intra-class distance and maximizing the inter-class distance, retaining the natural variation within the class, and taking into account both similarity and difference. 2 In the DA type I and type II algorithms, the discriminant matrix is ​​determined according to different optimization criteria, and the balance factor λ is adjusted adaptively. kAutomatically balance the weights of “same-class differences” and “different-class distinctions” to improve the system’s adaptability and training efficiency. Compared to 2DLPP, which ignores expression change information and 2DDLPP, which assumes that samples are in the same manifold space and ignores changes in similar samples, TDM 2 DA can more effectively capture the changing information between expressions, avoid the "overfitting" problem, more comprehensively explore expression features, improve the ability to distinguish different expressions, and thus improve the accuracy of expression recognition.

[0038] (3) Through experimental comparison on standard expression databases such as Cohn-Kanade library and RML library, the performance of this method is compared with that of various 1D algorithms (such as PCA, LDA, etc.) and 2D algorithms (such as 2DPCA, 2DLPP, etc.). The experimental results show that TDM 2 DA has excellent recognition performance, TDM 2 The recognition rates of DA Type I and Type II on two expression databases were higher than those of numerous comparative algorithms, fully demonstrating the method's advanced capabilities in expression feature extraction and recognition, providing a more efficient and accurate technical solution for the field of expression recognition. Furthermore, by optimizing the process design, the present invention makes the facial expression recognition process more concise and efficient, improving the system's practicality and operability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] It should be noted that unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0042] like Figure 1 As shown in the figure, a two-dimensional multi-manifold discriminant analysis algorithm (Two Dimensional MultipleManifold DiscriminantAnalysis, TDM 2 The facial expression recognition method of DA comprises the following steps:

[0043] Step S1: extract multiple salient regions of the facial image using the active shape model, including the left eye, right eye, left cheek, right cheek, and mouth, classify them into an expression database, and adjust the salient regions of each sample to a uniform size to form a set of salient regions of the same size;

[0044] Step S2: using a two-dimensional fractional Fourier transform to perform frequency domain conversion on each salient region so that the image is jointly expressed in the time-frequency plane;

[0045] Step S3: construct the set of salient regions of each expression category into an independent manifold to form a multi-manifold model, whose overall dimension is the product of the dimension of a single salient region and the number of salient regions;

[0046] Step S4, designing an objective function, including a manifold variation information maximization item, a manifold similarity information minimization item, and an inter-manifold edge information maximization item, wherein: the manifold variation information maximization item is calculated by weighting the local neighbor differences of similar samples; the manifold similarity information minimization item is calculated by weighting the local neighbor similarities of similar samples; and the inter-manifold edge information maximization item is calculated by weighting the local neighbor differences of heterogeneous samples;

[0047] Step S5: optimizing the objective function based on the difference criterion or the entropy criterion, and calculating the discrimination matrix corresponding to each type of expression;

[0048] Step S6: Project the salient area of ​​the test sample into a low-dimensional space, and use the discriminant matrix to perform facial expression classification.

[0049] The following is a detailed description of the implementation process of each step:

[0050] The specific implementation process of step S1 is:

[0051] Firstly, the Active Shape Model (ASM) method is used to extract the salient regions of each facial image. There are five salient regions in total, including the left eye, right eye, left cheek, right cheek and mouth. The size of each salient region is adjusted to a×b.

[0052] Secondly, from each expression sample Extract five salient regions from (the i-th sample of the k-th expression) to form a set of salient regions of the same size And the size of each salient region is a×b.

[0053] Then, construct the expression manifold based on the set of salient regions and define M k represents the k-th expression manifold, where l k =5·n k .

[0054] Step S2: First, perform a one-dimensional discrete fractional Fourier transform (FrFT) process. For the one-dimensional signal f(u′), perform a p-order fractional Fourier transform process. The result after the transformation is recorded as fp(u) , the calculation formula is as follows:

[0055] Where, is an imaginary unit; Indicates the angle of rotation in the time-frequency plane; p is the transformation order of FrFT, and p≠2n, n is an integer; F p represents the p-order fractional Fourier transform operator; f p(u) K represents the function of variable u obtained by processing the function f(u′) by p-order fractional Fourier transform; p (u,u′) is the corresponding kernel function.

[0056] Then, a two-dimensional fractional Fourier transform (2D-FrFT) is performed according to formula (2), specifically using the following discrete form:

[0057] Where x(p,q) is the original image, p and q are the row and column indices of the original image respectively; the image size is M×N; is the output image obtained after two-dimensional fractional Fourier transform, where m and n are the row and column coordinates of the output image respectively; is the two-dimensional fractional Fourier kernel function.

[0058] Equivalently, the two-dimensional FrFT transform can be decomposed into two one-dimensional FrFT operations, expressed as:

[0059] Where, is the fractional-order kernel function in the horizontal direction, is the fractional-order kernel function in the vertical direction. The fractional-order parameter is set to p1 = p2 = 0.5, which means that both are 1 / 2-order fractional Fourier transforms.

[0060] Step S3: After 2D-FrFT preprocessing, the salient regions of each expression category are constructed as independent manifolds to form a multi-manifold model. Specifically, each group of salient regions is constructed as a manifold, with each manifold corresponding to a specific expression category. By integrating all salient regions representing different expressions, an overall multi-manifold model is formed, whose overall dimension is the product of the dimension of a single salient region and the number of salient regions.

[0061] The specific implementation process of step S4 is: using TDM 2 DA extracts c discriminant matrices from the expression database in step S3, and then projects each salient region into the corresponding low-dimensional space, such as and v=1,…,l k , so that under a certain criterion Able to express in d k Represents the dimension of the low-dimensional space.

[0062] In order to represent the relationship between salient regions, let G = (Y, E) be a weighted undirected graph, where Y represents the vertex set corresponding to all salient regions and E represents the edge information connecting the salient regions. 2 The objective function of DA is:

[0063] Where W k is the discrimination matrix corresponding to the k-th expression, max J d (W k ) represents the maximization of the change information within the manifold, min J w (W k ) represents the minimization of similarity information within the manifold, max J b (W k ) represents the maximization of edge information between manifolds, is the vth significant region feature vector in the kth expression manifold, in formula (a) express The closest k in the same expression manifold d Nearest neighbor samples, in formula (b) express k in the same expression manifold w similar neighbor samples, in formula (c) express The nearest k in heterogeneous expression manifolds b Nearest neighbor samples, express and The change weight between express and The similarity weight between express and The boundary clarity weight between For K with similar expressions d Neighbor set, For K with similar expressions w Neighbor set, For K with different expressions b The nearest neighbor set, k d 、k w 、kb is the corresponding number of selected neighbors.

[0064] From formula (4), we can see that J d (W k ) and J w (W k ) respectively reflect the change information and similarity information within the manifold, while J b (W k ) reflects the edge information between manifolds.

[0065] That is, the maximization formula (4a) maximizes the difference in changes between similar samples in the local neighborhood, strengthens the expressive ability of key change features between similar samples, that is, enhances the contribution of local structure to category distinction, and is used to enhance the fine-grained recognition ability of the model; the minimization formula (4b) makes the salient areas within the same expression manifold closer after projection, ensuring the compactness of the same expression features; the maximization formula (4c) makes the salient areas of different expression manifolds farther apart after projection, enhancing the distinguishability between different expressions.

[0066] Step S5: Optimize the objective function based on the difference criterion or entropy criterion, and calculate the discrimination matrix corresponding to each type of expression. Combining formulas (4a), (4b) and (4c), TDM 2 The joint optimization form of DA can be divided into two optimization methods, as follows:

[0067] Difference criteria:

[0068] Entropy criterion:

[0069] Where λ k is the balancing factor, and its value range is [0,1]. c is the number of expression categories in the expression database, and k = 1, 2,…, c.

[0070] From equations (5) and (6), we can see that λ k The smaller it is, the greater the proportion of intra-class change information. k =1, it means that the change information between samples within the class is not considered.

[0071] Construct the local relationship matrix within and between manifolds to preserve the change information between samples within the manifold. and They are and If two vertices are very close, the change information of these two vertices is small and the similarity is large.

[0072] Where κ is the scale parameter and ||·|| is the Euclidean norm. is the vth significant region feature vector in the kth expression manifold, is the rth local neighbor sample with the largest difference in the same expression manifold, is the rth local neighbor sample with similar structure in the same expression manifold.

[0073] According to formula (7) and formula (8), J w (W k ) and J d (W k ) can be expressed by formula (9) and formula (10) respectively:

[0074] Where, and and They are called the intra-manifold similarity matrix and the variation matrix respectively. It can be seen that they are symmetric and semi-positive definite. For convenience, the above two matrices are simplified to Equations (11) and (12):

[0075] Where, and and They are all diagonal matrices, and their entities are and The sum of the column vectors of and and are diagonal matrices, and their entities are and The sum of the row vectors of and

[0076] Then, maximize the distance between manifolds. For any vertex Used to calculate samples and The sum of the Euclidean distances between . Therefore, J b (W k ) explains and The geometric structure information between them.

[0077] max{J b (W k )}Try to and The distance between them is the largest, that is, the heterogeneous samples are as far away as possible. In addition, in order to make TDM 2 DA is more robust to marginal samples. and When they are very close, and A relatively large penalty factor; on the contrary, when and When far apart, give and Small penalty factor. Based on the above principles, Defined as:

[0078] Where κ is the scale parameter, ||·|| is the Euclidean norm, is the vth significant region feature vector in the kth expression manifold, is the rth local neighbor sample close to the boundary in the heterogeneous expression manifold.

[0079] From this we can get,

[0080] Where, is an entity l k ×(k b *l k )matrix, and is a diagonal matrix, and its entities are The rows and columns of j=1,…,k b l k .

[0081] Based on two-dimensional multi-manifold discriminant analysis (TDM 2 There are two implementation forms of the facial expression feature extraction method of DA. The execution steps of the two implementation forms are given below:

[0082] (1)TDM 2 DAI type:

[0083] J b (W k ), J d (W k ) and J w (W k ) into the optimization objective function (5), we can get the reduced result, as shown in (16).

[0084] However, there is no approximate optimal solution for formula (13) that can obtain c discriminant matrices at the same time. In order to solve this problem, based on the feature extraction idea of ​​MSD, the discriminant matrix of each category is obtained one by one.

[0085] because as well as We can get λ k The estimated value of

[0086] Next, for the discriminant matrix W in formula (2) k Dimension d k The selection method of d needs to be considered. k The traditional feature extraction algorithm only involves one discriminant matrix, while multi-manifold discriminant analysis requires the extraction of c discriminant matrices. The traditional method is to use genetic algorithms to approximately search for the optimal dimension d. k However, the use of genetic algorithms requires pre-learning of training samples, which consumes a lot of time. This embodiment uses the feature dimension selection method under the trace ratio to calculate the discriminant matrix dimension of each category of expression. From formulas (11), (12) and (15), we can see that as well as are respectively positive semi-definite matrices, we can get:

[0087] Where i = 1,…,d k .

[0088] In summary, TDM 2 The specific steps of the DAI algorithm are shown in Table 1: Table 1 TDM 2 Specific steps of the DAI algorithm

[0089] (2)TDM 2 DAII type:

[0090] J b (W k ), J d (W k ) and J w (W k ) is substituted into (6) to obtain the TDM under the trace ratio 2 DA optimization criteria:

[0091] Formula (19) can be rewritten as:

[0092] Due to λ k ∈[0,1], and and They are all positive semidefinite matrices, so we know that J 3 (W k )≥J 2 (W k ). Available from J 3 (W k ) and find the discriminant matrix corresponding to each manifold in turn.

[0093] In summary, TDM 2 The specific steps of the DAII algorithm are shown in Table 2: Table 2TDM 2 Specific steps of the DAII algorithm

[0094] It is worth noting that the trace ratio optimization problem is often transformed into a trace ratio problem, that is, the optimization problem of finding the following formula: Similar to the difference criterion, the trace problem also requires the prior determination of the dimension of each type of discriminant matrix. The eigenvalues ​​corresponding to the eigenvectors of In order to maximize formula (20), choose d k feature vectors, such as but The above choice means that the inter-manifold distance of the expression is greater than the intra-manifold distance, and along The direction of maximally retains the edge information between manifolds.

[0095] The implementation process of step S6 is as follows: set the category label of the test sample, extract the significant area and perform frequency domain conversion on the test sample, load the discriminant matrix obtained through training, perform projection operation to form the final low-dimensional representation, and calculate the expression category information of the test sample, that is, find the expression category with the smallest distance from the test sample as its classification result.

[0096] To compare TDM 2 The performance of DA and other expression feature extraction methods, the parameters of the simulation experiment are set as follows: considering the sample size in the expression library, the size of each significant block in the Cohn-Kanade library and the RML library is set to 32×32 and 16×16 respectively; in order to avoid the "small sample" problem of LDA, PCA is used to reduce the dimension of the training samples and keep the dimension as nc; for the MMSD method, the balance factor σ is set to trace(S b ) / trace(S w); For 1D and 2D manifold learning methods, determining the number of neighbors is still an open problem. Therefore, the value is verified by searching, that is, gradually increasing the number of inter-class neighbors from 20 to k 2 l-1, with an interval of 10; gradually increase the number of intra-class neighbors from 10 to n×k, with an interval of 4; for SLPP, set the additional factor α to 1; for Xiao's method, first divide the training set into two parts: parameter training and parameter testing. Specifically, 50% of the expression samples are selected for training manifold learning, 25% for parameter adjustment, and the remaining 25% for testing. To fairly compare the performance of each algorithm, the feature dimension is selected to achieve the best recognition performance.

[0097] The experimental comparison results of expression recognition performance of different algorithms are shown in Table 3. As can be seen from Table 3:

[0098] 1D vs. 2D Algorithms: 1D algorithms require converting each sample into a one-dimensional matrix representation, which can result in loss of original facial expression structure information and increase computational complexity. More seriously, 1D algorithms can face the "small sample size" problem. To address this, additional factors, such as perturbation factors, may be needed to weaken discriminative capabilities. Therefore, in most cases, 2D algorithms offer superior expression recognition performance compared to 1D algorithms, for example: 2DPCA > PCA, and 2DLPP > LPP.

[0099] TDM 2 The experimental comparison results of DA and 2DPCA algorithms show that: TDM 2 DA demonstrates significant advantages over 2DPCA, 2DLDA, and 2DMMC. This is because these three algorithms cannot effectively overcome the effects of facial pose, lighting, and other factors on facial expressions. In particular, they assume that facial regions share equally important information about facial expressions, but lack information describing the local geometric structure of facial expressions.

[0100] TDM 2 The experimental comparison results of DA, 2DLPP and 2DDLPP showed that TDM 2 DA performs better than 2DLPP and 2DDLPP. 2DLPP only considers the similarity information between neighboring samples, but ignores the change information between different expressions, and it is an unsupervised algorithm; 2DDLPP maximizes the distance between manifolds while minimizing the distance within the manifold, and its recognition performance is better than 2DLPP. However, 2DDLPP assumes that all samples are in the same manifold space, and also ignores the change information between similar samples, which may lead to the emergence of "overfitting" problems. In contrast, TDM 2DA can more effectively capture the changing information between expressions, thereby improving the recognition performance.

[0101] TDM 2 The experimental comparison results of DA, 2DSLSDP and 2DNMVE show that: TDM 2 DA performs better than 2DSLSDP and 2DNMVE. 2DSLSDP retains the variation information between samples of the same type, attempts to minimize the similarity between samples of the same type while maximizing the difference information between heterogeneous samples. However, it is difficult or almost impossible for 2DSLSDP to satisfy the above three constraints at the same time in a manifold space. For 2DNMVE, it claims that inter-class and intra-class variation information should not be treated equally, and thus obtains better expression recognition results than 2DSLSDP. However, 2DNMVE also has problems that 2DSLSDP cannot overcome, which results in its recognition performance not being better than TDM. 2 DA is better. Table 3 Comparison of recognition performance of different algorithms in Cohn-Kanade and RML libraries (%)

[0102] TDM 2 Both DA implementation algorithms outperform other compared algorithms. This is because TDM 2 DA not only maintains the difference information between samples in the same class in the space after dimensionality reduction, but also minimizes the distance between the same manifolds and maximizes the distance between different manifolds, so it can effectively extract the identification information of specific expressions rather than the identification information of specific people's expressions. It should be noted that TDM 2 The recognition performance of DAI type is better than TDM 2 DAII type, because the difference criterion algorithm is better than the entropy criterion. The difference criterion is more inclined to solve small sample problems, while the entropy criterion is superior for large sample problems.

[0103] The present invention analyzes the facial salient areas and TDM 2 The application of the DA algorithm can effectively extract the discriminant matrix for various facial expressions and achieve accurate recognition of facial expressions. Compared with traditional methods, this method considers the saliency of facial regions and fully utilizes the similarity between samples of the same type, the edge information between different samples, and the variation information between samples of the same type. This improves the robustness and generalization ability of recognition, and has broad application prospects and economic value.

[0104] Finally, it should be noted that the parts of the present invention that are not described in detail are all prior art. Those skilled in the art will understand that the above description is only a preferred embodiment of the invention and is not intended to limit the invention. Although the invention has been described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions described in the above examples or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims

1. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm, characterized in that: The following steps are involved: Step S1: extract multiple salient regions of the facial image using the active shape model, including the left eye, right eye, left cheek, right cheek, and mouth, classify them into an expression database, and adjust the salient regions of each sample to a uniform size to form a set of salient regions of the same size; Step S2: using a two-dimensional fractional Fourier transform to perform frequency domain conversion on each salient region so that the image is jointly expressed in the time-frequency plane; Step S3: construct the set of salient regions of each expression category into an independent manifold to form a multi-manifold model, whose overall dimension is the product of the dimension of a single salient region and the number of salient regions; Step S4, designing an objective function, including a manifold variation information maximization item, a manifold similarity information minimization item, and an inter-manifold edge information maximization item, wherein: the manifold variation information maximization item is calculated by weighting the local neighbor differences of similar samples; the manifold similarity information minimization item is calculated by weighting the local neighbor similarities of similar samples; and the inter-manifold edge information maximization item is calculated by weighting the local neighbor differences of heterogeneous samples; Step S5: optimizing the objective function based on the difference criterion or the entropy criterion, and calculating the discrimination matrix corresponding to each type of expression; Step S6: Project the salient area of ​​the test sample into a low-dimensional space, and use the discriminant matrix to perform facial expression classification.

2. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 1, characterized in that: The order parameters P1 and P2 of the two-dimensional fractional Fourier transform are both set to 0.

5.

3. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 2, characterized in that: The specific implementation process of step S2 is: Step S21: perform one-dimensional discrete fractional Fourier transform according to the following formula: Where, is an imaginary unit; Indicates the angle of rotation in the time-frequency plane; p is the transformation order of FrFT, and p≠2n, n is an integer; F p represents the p-order fractional Fourier transform operator; f p(u) K represents the function of variable u obtained by processing the function f(u′) by p-order fractional Fourier transform; p (u,u′) is the corresponding kernel function; Step S22: perform a two-dimensional fractional Fourier transform using the following discrete form: Where x(p,q) is the original image, p and q are the row and column indices of the original image respectively; the image size is M×N; is the output image obtained after two-dimensional fractional Fourier transform, where m and n are the row and column coordinates of the output image respectively; is the two-dimensional fractional Fourier kernel function.

4. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 2, characterized in that: The mathematical expression of the objective function in step S4 is: Where W k is the discrimination matrix corresponding to the k-th expression, max J b (W k ) represents the maximization of the change information within the manifold, min J w (W k ) represents the minimization of similarity information within the manifold, max J b (W k ) represents the maximization of edge information between manifolds, is the vth significant region feature vector in the kth expression manifold, in formula (a) express The closest k in the same expression manifold d Nearest neighbor samples, in formula (b) express k in the same expression manifold w similar neighbor samples, in formula (c) express The nearest k in heterogeneous expression manifolds b Nearest neighbor samples, express and The weight of the change between express and The similarity weight between express and The boundary clarity weight between For K with similar expressions d Neighbor set, For K with similar expressions w Neighbor set, For K with different expressions b The nearest neighbor set, k d 、k w 、k b is the corresponding number of selected neighbors.

5. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 4, characterized in that: The optimization objective function of the difference criterion in step S5 is: Where λ k is the balancing factor, and its value range is [0,1]. c is the number of expression categories in the expression database, and k = 1, 2,…, c.

6. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 4, characterized in that: The optimization objective function of the entropy criterion in step S5 is: Where λ k is the balancing factor, and its value range is [0,1]. c is the number of expression categories in the expression database, and k = 1, 2,…, c.

7. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 5 or 6, characterized in that: The balance factor λ k Calculated by matrix trace ratio: Where, and are the similarity matrix and change matrix within the manifold, is the edge matrix between manifolds, and trace(·) represents the trace of the matrix.

8. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 4, characterized in that: In step S5, the discriminant matrix W is solved by the eigenvalue decomposition method. k , select the eigenvectors that satisfy the following conditions: Where, and are the similarity matrix and change matrix within the manifold, is the edge matrix between manifolds; trace(·) represents the trace of the matrix; is the i-th projection direction of the k-th expression, λ is the balance factor, and a is the height of the salient area.

9. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 4, characterized in that: and The calculation formula is as follows: Where κ is the scale parameter, ||·|| is the Euclidean norm, is the vth significant region feature vector in the kth expression manifold, is the rth local neighbor sample with the largest difference in the same expression manifold, is the rth local neighbor sample with similar structure in the same expression manifold, is the rth local neighbor sample close to the boundary in the heterogeneous expression manifold.

10. A facial expression recognition method based on a two-dimensional multi-manifold discriminant analysis algorithm according to claim 1, characterized in that: The low-dimensional feature dimension d after projection in step S6 k It is dynamically determined by the trace ratio criterion to ensure that the maximum inter-manifold edge information is preserved.

Images