A method for facial depression expression recognition based on machine learning
Through the domain adaptation of depression recognition model, using robust latent regression functions and regularization terms, the subspace learning and semi-supervised framework is integrated, and the problems of insufficient data marker volume and environmental differences in traditional methods are solved, achieving high-accuracy facial Depressive expression recognition.
Patent Information
- Application Number
- CN202111512514.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-08
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-12-08
AI Technical Summary
Traditional machine learning requires a large amount of labeled data in facial depression expression recognition, and the differences in different experimental subjects and environments lead to poor classification effects, making it difficult to achieve accurate depression recognition.
The domain adaptation depression recognition model is adopted, and the potential spatial relationship between the target video set and the source video set is constructed through robust latent regression functions and regularization terms. The pseudo-class label matrix and lσ norm are used to integrate subspace learning and semi-supervised fields. Adapt to the framework and share discriminant information from different domains.
It improves the accuracy and robustness of facial image depression recognition, is suitable for facial appearance and dynamic characteristics under different conditions, and the depression recognition performance in cross-domain scenarios is significantly improved.
Smart Images

Figure CN114419479B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of emotion recognition, and in particular to a facial depression expression recognition method based on machine learning. Background Art
[0002] In order to detect patients with depression tendencies as early as possible, it is useful to detect their emotions. Human emotions can be identified in many ways, such as electrocardiogram, electroencephalogram, speech, facial expressions, etc. Among various emotional signals, physiological signals are widely used in emotion recognition.
[0003] Traditional machine learning for vision-based ADD requires sufficient labeled training data to learn a classification model for evaluating facial images in an unlabeled test dataset, which is assumed to have the same distribution as the training dataset. Although supervised learning techniques have made great contributions to machine learning-based depression recognition, their performance is often limited by the amount of available labeled training data, while labeling large amounts of data is expensive and time-consuming. In addition, existing ADDs typically achieve accurate depression recognition by tailoring a classifier for each subject. Training a classifier for a specific subject using limited data collected from multiple subjects will often result in poor accuracy, as different subjects and shooting environments can affect the classification effect, including differences in pose, scale, lighting, occlusion, and individual facial shape, texture, and behavior. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a facial depression expression recognition method based on machine learning, which can accurately recognize depression expressions.
[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a facial depression expression recognition method based on machine learning, comprising the following steps:
[0006] (1) Obtain the target video set;
[0007] (2) Inputting the target video set into a domain-adaptive depression recognition model to obtain a final depression score, wherein the domain-adaptive depression recognition model includes a feature extraction module, a first target classifier and a second target classifier, the feature extraction module is used to extract features from the target video set to obtain expression features and dynamic features, the first target classifier is input with the expression features, the second target classifier is input with the dynamic features, a domain-adaptive relationship is established between the first target classifier and the source classifier, and the average values of the predicted data sets of the first target classifier and the second target classifier are close; the domain-adaptive depression recognition model is expressed as Among them, Ω LR(·) is the robust potential regression function; Loss(·) is the least square loss function, f t (·) is the second target classifier, f(·) is the first target classifier, W t ,W s and are model parameter matrices, is a dynamic feature, X t is the target video set, P is the latent space between the target video set and the original video set, F is the pseudo-class label matrix, X s is the source video dataset, Y s is the label matrix of the source video data, I r is the r-dimensional identity matrix, I n is the n-dimensional identity matrix, Ω DC (P) is the first regularization term, Ω(W) is the second regularization term, W = [W s ,W t ].
[0008] When designing the robust latent regression function, the low-level representation and high-level semantics of the video are associated through a decision function, the source video set and the target video set are correlated in the learned latent space by minimizing the constrained distribution difference, the optimal latent space is found by norm minimization and a linear regression mapping is learned from the feature matrix to its corresponding label matrix containing several different categories, and the target data is associated with the predicted label by utilizing a source classifier in the latent space.
[0009] The robust latent regression function Among them, μ is the balance coefficient, Y t represents the label matrix of the target video data, U represents the diagonal matrix, Indicates that in the target video set X t The normalized graph Laplacian matrix defined above, ||·|| σ Indicates l σ norm,||·|| F represents the Frobenius norm, and tr(·) represents the trace of the matrix.
[0010] The first regularization item uses the method of aligning the target video dataset and the source video dataset to achieve the approximation of the average values of the predicted datasets of the first target classifier and the second target classifier.
[0011] The second regularization term explores the shared information between the first target classifier and the source classifier in a manner of minimizing the tracking norm of W so that the rank reaches a maximum value.
[0012] Beneficial Effects
[0013] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art: The present invention integrates subspace learning and vision-based depression recognition into a unified learning framework by utilizing source domain knowledge and the correlation of models in different domains. Both the source and target loss functions are included in the various parts of the objective function, and the common components of different classifier models are encoded into the present method as low-rank regularization terms to utilize the discriminant information shared by different domains. In addition, the present invention also 2,1 The norm is introduced into the loss function to make the LDADR method robust to noise. The present invention transforms the depression recognition problem of learning classifiers in different domains into a semi-supervised domain adaptation framework, evaluates the performance of the proposed method in a cross-domain scenario involving facial appearance and dynamic features collected from different conditions, and experimentally verifies that LDADR has excellent performance in depression recognition based on facial images. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a block diagram of a domain-adapted depression recognition model in an embodiment of the present invention. DETAILED DESCRIPTION
[0015] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.
[0016] The embodiment of the present invention relates to a facial depression expression recognition method based on machine learning, comprising the following steps: obtaining a target video set; inputting the target video set into a domain adaptation depression recognition model to obtain a final depression score, wherein the domain adaptation depression recognition model comprises a feature extraction module, a first target classifier and a second target classifier, the feature extraction module is used to extract features from the target video set to obtain expression features and dynamic features, the input of the first target classifier is the expression features, the input of the second target classifier is the dynamic features, a domain adaptation relationship is established between the first target classifier and a source classifier, and the average values of the predicted data sets of the first target classifier and the second target classifier are close.
[0017] Specifically, in this embodiment, lowercase letters and uppercase bold letters are used to represent column vectors and matrices, respectively, for example and where n and d are the number of data and its dimension respectively. The (j,i) entry of the matrix A is represented by A j,i Indicates that its j-th row and i-th column are respectively composed of A j,: and A :,iThe transpose of a vector or matrix is represented by the superscript T, and the trace of a matrix A is represented by tr(A). n represents the identity matrix of size n×n, 1 d represents a d-dimensional column vector with all elements set to 1. The representation of merging k matrices along the row (horizontally) is [A 1 ,A 2 …,A k ], merging them along the column (vertically) is represented as [A 1 ; A 2 ;…;A k ]. Two matrices A 1 and A 2 The inner product of
[0018] Definition 1. The p-norm of a vector a is defined as The Frobenius norm of the matrix A, l 2,1 norm and trace norm are defined as
[0019] Definition 2. l of matrix A σ The norm is defined as Among them, σ is the adaptive parameter of the loss function. σ The norm has the following properties:
[0020] (1)l σ The norm is non-negative and convex, which is desirable for the loss function;
[0021] (2)l σ The norm can achieve quadratic differentiation;
[0022] (3) When but
[0023] (4) When Then ||A|| σ →(1+σ)||A|| 2,1 ;
[0024] (5) When σ→0, then ||A|| σ →||A|| 2,1 ;
[0025] (6) When σ→∞, then
[0026] In this implementation, the source domain and the target domain are defined by the subscripts “s” and “t” respectively. s The labeled source domain data of facial appearance features (denoted by A) After nonlinear mapping, the source video set is represented as And the label matrix is represented as in is the label vector and c is the number of classes. Marked as y i =j, then y ij =1, otherwise y ij = 0. In addition, s n with the same characteristics t Available set of target domain data Indicates that represents the target video set, is the corresponding label matrix. In this embodiment, the target training video and facial dynamic features (denoted by D) are expressed as Among them, d z is the feature dimension, n t is the amount of training data.
[0027] In this embodiment, respectively define: X = [X s ,X t ],Y=[Y s ,Y t ],n=n s +n t . Represented as a feature transformation matrix. Given a dataset Construct an undirected graph and use the weighted adjacency matrix M = [M i,j ] i,j=1,2,...,n Indicates that M i,j >0 means x i and x j Connect, otherwise M i,j = 0. The edge weight is determined as follows:
[0028]
[0029] Among them, δ k (x) is the set of k nearest neighbors of x, and γ is the bandwidth parameter, which can be chosen empirically. where θ is the square root of the mean norm of the training data X. Define the diagonal matrix and its diagonal elements Then the graph Laplacian L can be written as This embodiment uses the normalized graph Laplacian operator, which is defined as
[0030] This implementation introduces a pseudo-class label matrix to enhance the robustness of the framework. Define the pseudo-class label matrix of the data in the target domain, where f iis x i The corresponding pseudo-class tag. T =(Y t (Y t ) T ) -1 / 2 Y t (Y t ) T (Y t (Y t ) T ) -1 / 2 =I n , impose the following constraints on F: FF T =I n , F ≥ 0. Therefore, the j-th column of F is obtained as follows: where n j is the number of data in the jth class.
[0031] In the vision-based depression recognition task, it is assumed that there is a large amount of labeled source data and a very small amount of labeled target data. The goal of this embodiment is to find a common latent space P to assist the depression recognition task in the target domain with scarce labels by leveraging the knowledge in the source domain with rich labels. The new feature representation in this space can minimize the mismatch problem of domain distribution while preserving the structural properties of the original data. This embodiment sets an orthogonal constraint on P so that P T P=I r , where r(<d) is the dimension of the shared latent space. In addition, to address the problem of insufficient target labeled data, this embodiment will use the assumption that unlabeled target data should be encoded smoothly in the regularizer.
[0032] To this end, this embodiment proposes a latent domain adaptation depression recognition (LDADR) framework by finding a joint latent space and semi-supervised DA learning combined with facial appearance and dynamic feature representation. The method is based on the assumption that different but related learning models have some common structures. This embodiment reveals the potential differences and commonalities between two different domains by learning a feature representation subspace. Therefore, two classification functions of different domains are learned simultaneously in a unified framework, allowing the final algorithm to utilize the knowledge of the two learning models as complementary information. The common components of different classification functions can be encoded by regularization terms. Specifically, the objective function of LDADR consists of four parts, namely, robust regression in the latent space, structure preservation within and across domains, discriminant regularization of predicted labels, and relevance of domain classification models. The first two aim to explore the invariant low-dimensional structure across domains while minimizing the structural risk of the learning model on the latent space. The latter two mine the intrinsic discriminant information in the target domain and further exploit the discriminant structure of the source domain. After obtaining the prediction function of the latent space, the label of the new target instance can be determined.
[0033] Figure 1 Schematic diagram of the domain-adapted depression recognition model in this embodiment. The KPCA nonlinear mapping is applied to uniform features (e.g., facial appearance) of the source / auxiliary and target videos by representation "A". The source depression classifier and the target depression classifier obtained from the uniform features may have common components containing irrelevant and noisy. The present embodiment eliminates this negative information by jointly optimizing the source and target depression classifiers with representation "A", thereby bringing discriminative knowledge to depression recognition. On the other hand, the present embodiment has heterogeneous feature representations "D" (e.g., facial dynamics) from facial videos and combines them with Figure 1 Therefore, another target depression classifier "AD" will be trained based on the result representation of the nonlinear mapping from facial appearance (denoted by "A") and dynamics (denoted by "D"). This embodiment will integrate and optimize the two depression classifiers, and finally merge the decision values of the two into the final depression score.
[0034] The basic expression of the domain adaptation depression recognition model (LADAR) in this implementation can be uniformly expressed as: Among them, Ω LR (·) is the robust potential regression function; Loss(·) is the least square loss function, f t (·) is the second target classifier (i.e., target depression classifier D), f(·) is the first target classifier (i.e., target depression classifier A), W t ,W s and are model parameter matrices, Ω DC (P) is the first regularization term, which can be used to align the local structure within the domain and the distribution difference between domains. Ω(W) is the second regularization term, W = [W s ,W t ], using this regularization term to encode the domain classification model W = [W s ,W t ]’s relevance.
[0035] When designing a robust latent regression, first, the low-level representation of the video and the high-level semantics are associated through the decision function f(·), such as outputting the predicted label y for the input video sequence z. Define the multi-category decision function f in the target domain t (·)for in is a depression classifier model, which is associated and the pseudo class label matrix F.
[0036] Next, the source video dataset and the target depression video dataset have different experimental subjects but are related and are represented only by similar features, that is, A assists in assisting the target depression recognition with a small number of labeled samples. Since the source domain and the target domain are related in the learned latent space by minimizing the constrained distribution difference, this embodiment assumes that the two domains will have some shared knowledge, such as discriminative structure. In addition, this embodiment also assumes that the target domain will retain the discriminative structure in the original space in the new latent space. To this end, this embodiment defines the multi-class decision function of the target domain as f(P,X t ,W t ,W s )=(X t ) T (W t +PW s ). Similarly, the multi-class decision function of the source domain can be defined as f(P,X s ,W s )=(X s ) T PW s .
[0037] Then use l σ The robust latent regression problem is solved by minimizing the norm of the latent space P, in which the optimal latent space P is found and a linear regression mapping from the feature matrix to its corresponding label matrix containing c different categories is learned. σ The regression formula of the norm projects the features of the source domain into semantic labels in the latent space:
[0038] Loss(f(P,X s ,W s ),Y s )=||(X s ) T PW s -Y s || σ .(3)
[0039] In addition, we will search for the target depression classifier W in the original space t , to associate the target data with their predicted labels F by leveraging the source classifier W in the latent space:
[0040] Loss(f(P,X t ,W t ,W s ),F)=||(X t ) T (W t +PW s )-F|| σ +tr((FY t )T U(FY t )). (4)
[0041] in is a diagonal matrix whose diagonal elements U i,i Defined as: If x i ∈X is the labeled data, then U i,i ≥0, otherwise U i,i =0. In this embodiment, simply set U i,i = 1. The second term in formula (4) is a fitting constraint, the purpose of which is to make the classification function ensure that the predicted label matrix is consistent with the initial label assignment.
[0042] By making X t W t The predicted labels obtained are the same as those in On The predicted labels obtained are consistent, so and W t More precise. In this way, heterogeneous features from the target and W from knowledge adaptation t Together, they will enhance the accuracy of depression identification. to achieve this.
[0043] This implementation incorporates the inherent data geometry into the predicted pseudo-labels to address the problem of insufficient supervision information in the target domain. This method helps propagate existing labels between unlabeled data in the target domain and keeps the predicted labels consistent with the data manifold structure. The local preservation objective function is usually defined as
[0044]
[0045] in, Indicates that the target data The normalized graph Laplacian matrix defined above.
[0046] Finally, the latent regression function is defined as follows.
[0047]
[0048] Among them, μ is the balance coefficient, which is used for the balancing effect.
[0049] When designing the first regularization term, this embodiment preserves the local geometric structure of the domain data through a formula commonly used in manifold learning:
[0050]
[0051] Although the correlation constraint of the discriminant structure in equation (7) has advantages over traditional methods, it does not explicitly model the relationship between domains. Therefore, this embodiment further introduces a global distribution consistency regularizer to ensure the correlation between domains. A direct method is to make the means of the two domain data very close, that is, to minimize the following problem:
[0052]
[0053] The empirical maximum mean difference (MMD) actually compares various distributions by projecting the average distance of samples in two domains under the feature space, that is:
[0054]
[0055] The MMD strategy expressed in the above formula can reduce the difference in marginal distributions, but it does not approach the conditional distribution divergence of the two domains. In classification problems, the conditional distribution mismatch between two different domains must be reduced. When the target sample is completely unannotated, the alignment of the conditional distribution becomes very important. To this end, this embodiment adopts a probabilistic classification adaptation formula to effectively guide the transfer of intrinsic knowledge. In this way, the predicted label of the target sample can also benefit the domain alignment.
[0056] In order to promote the use of labels in multiple categories and thus address the conditional distribution differences across the two domains, this implementation introduces probabilistic labels into MMD modeling and designs a novel weighted category adaptive loss function as follows:
[0057]
[0058] in,‖·‖ F represents the Frobenius norm, represents the source sample size of class c. represents the target sample size of the cth class. Approximate calculation N s / t ∈R C×C is a diagonal matrix, where the cth diagonal element is In fact, by considering the prior category distribution, the above formula can resist the influence of class weight bias.
[0059] Based on the above formula and projection alignment, we have the goal:
[0060]
[0061]
[0062] exist When it is a matrix of all 1s, it is represented as a center matrix. Such constraints will help maintain the variance of the data after adaptation, which further brings additional data discrimination ability during the learning process.
[0063]
[0064] Among them, the matrix
[0065] The above regularizer has shown excellent results in the research. Therefore, this embodiment uses the regularizer to align two data sets in two different domains. The regularizer will force the average values of the two predicted data sets to be close. Therefore, combining equation (8) with equation (7) can obtain the following global and local distribution consistent regularization function.
[0066]
[0067] When designing the second regularization term, since the two domains are related, the source domain video set and the target domain video set are also related at the model level. In this embodiment, when the same type of features are used to represent videos from the two domains, they will have some shared components. Therefore, this embodiment adds the regularization term Ω(W) to the objective function to explore the shared information between the classification functions of the two different domains.
[0068] The shared structure of the two model parameters can be expressed as their low-rank matrices. This embodiment assumes that the common information of the domain classification model can be shared by adding low-rank moment constraints. Since the rank of the minimized matrix is non-convex, this embodiment maximizes the rank by minimizing the tracking norm of W. On the other hand, the tracking norm term enables different classification functions to share common components / knowledge across domains. In this way, information from different domains can be transferred from one domain to another. Specifically, this embodiment defines the relevant regularization function of the classification model as:
[0069]
[0070] Combining equations (6), (9) and (10), we get the following minimization problem:
[0071]
[0072] The above optimization problem simultaneously learns the latent space P, the target prediction label matrix and the model parameters in the decision function W s and W t ,in
[0073] In this embodiment, the obtained W s ,P, and W t Applied to the test video for depression recognition, the corresponding decision values are obtained. The obtained decision values are then normalized and weighted combined to obtain the final decision value for the test video. The purpose of this approach is to: 1) adapt the knowledge from the source video to the target video by exploiting the shared structure between them; 2) enhance observations from specific features that are only possessed by the target depression recognition video.
[0074] It is not difficult to find that the present invention integrates subspace learning and vision-based depression recognition into a unified learning framework by utilizing source domain knowledge and the correlation of models in different domains. Both the source and target loss functions are included in the various parts of the objective function, and the common components of different classifier models are encoded into the present method as low-rank regularization terms to utilize the discriminant information shared by different domains. In addition, the present invention also 2,1 The norm is introduced into the loss function to make the LDADR method robust to noise. The present invention transforms the depression recognition problem of learning classifiers in different domains into a semi-supervised domain adaptation framework, evaluates the performance of the proposed method in a cross-domain scenario involving facial appearance and dynamic features collected from different conditions, and experimentally verifies that LDADR has excellent performance in depression recognition based on facial images.
Claims
1. A facial depression expression recognition method based on machine learning, characterized in that: The following steps are involved: (1) Obtain the target video set; (2) Inputting the target video set into a domain-adaptive depression recognition model to obtain a final depression score, wherein the domain-adaptive depression recognition model includes a feature extraction module, a first target classifier and a second target classifier, the feature extraction module is used to extract features from the target video set to obtain expression features and dynamic features, the first target classifier is input with the expression features, the second target classifier is input with the dynamic features, a domain-adaptive relationship is established between the first target classifier and the source classifier, and the average values of the predicted data sets of the first target classifier and the second target classifier are close; the domain-adaptive depression recognition model is expressed as Among them, Ω LR (·) is the robust potential regression function; Loss(·) is the least square loss function, f t (·) is the second target classifier, f(·) is the first target classifier, W t ,W s and are model parameter matrices, is a dynamic feature, X t is the target video set, P is the latent space between the target video set and the original video set, F is the pseudo-class label matrix, X s is the source video dataset, Y s is the label matrix of the source video data, I r is the r-dimensional identity matrix, I n is the n-dimensional identity matrix, Ω DC (P) is the first regularization term, Ω(W) is the second regularization term, W = [W s ,W t ]; The robust latent regression function is designed to associate low-level representations and high-level semantics of the video through a decision function, to cause the source video set and the target video set to be associated in the learned latent space by minimizing the constrained distribution difference, to find the best latent space by minimizing the norm and to learn a linear regression mapping from a feature matrix to its corresponding label matrix containing several different categories, and to associate the target data with the predicted label by utilizing a source classifier in the latent space; The robust latent regression function Among them, μ is the balance coefficient, Y t represents the label matrix of the target video data, U represents the diagonal matrix, Indicates that in the target video set X t The normalized graph Laplacian matrix defined above, ||·|| σ Indicates l σ norm,||·|| F represents the Frobenius norm, and tr(·) represents the trace of the matrix.
2. The method for recognizing facial depression expression based on machine learning according to claim 1, characterized in that: The first regularization item uses the method of aligning the target video dataset and the source video dataset to achieve the approximation of the average values of the predicted datasets of the first target classifier and the second target classifier.
3. The method for facial depression expression recognition based on machine learning according to claim 1, characterized in that: The second regularization term explores the shared information between the first target classifier and the source classifier in a manner of minimizing the tracking norm of W so that the rank reaches a maximum value.
