A facial expression recognition method based on multi-element fusion decision

Through the face expression recognition method of multi-variable fusion decision-making, the lightweight attention mechanism and multi-example learning and extracting features, and combined with the random forest decision tree model, the problem of inaccurate recognition effect in the existing technology is solved, achieving higher classification accuracy and recognition effect.

CN114429654BActive Publication Date: 2024-11-01NANJING NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111421431.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-26
Publication Date
2024-11-01
Estimated Expiration
2041-11-26

AI Technical Summary

Technical Problem

The prior art has the problem that the recognition effect is not accurate enough in facial expression recognition, especially in the feature extraction and classification recognition stages, which is difficult to meet the needs of more accurate.

Method used

The facial expression recognition method with multiple fusion decisions is adopted, and the face detection algorithm based on the lightweight attention mechanism is preprocessed, and feature extraction is performed in combination with multi-example learning and attention mechanism, and age, gender and skin color are introduced as weighted discrimination. Multi-factor, using a random forest decision tree model for expression classification, to achieve more accurate decision-making.

Benefits of technology

The classification accuracy and recognition effect of facial expression recognition are improved, more accurate basic elements are provided for classification models, the classification accuracy of expression classification based on multiple fusion is improved, and the effectiveness of the method is verified through expression recognition tools.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114429654B_ABST
    Figure CN114429654B_ABST
Patent Text Reader

Abstract

The present invention discloses a facial expression recognition method based on multivariate fusion decision-making, comprising the following steps: using a face detection algorithm based on a lightweight attention mechanism to perform positioning preprocessing on an input face image; based on the preprocessing positioning, performing facial feature extraction, optimizing the feature extraction with a unique weight ratio and a learning mechanism during the extraction process; based on the extracted feature information, outputting a classification decision basis with the aid of multivariate fusion; and classifying the expression of the object to be tested based on the feature information and multivariate fusion classification. The present invention solves the problem of improving the accuracy of face pre-positioning by integrating and jointly deciding on a variety of factors that affect facial expression recognition based on a random forest decision tree to achieve a better expression recognition effect, solves the problem of the accuracy of feature extraction according to the correlation between actual features and their multivariate causes, and improves the accuracy of expression classification based on multivariate fusion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and relates to facial image detection, facial feature extraction, and attention mechanism, and in particular to a facial expression recognition method with multi-element fusion decision-making. Background Art

[0002] Facial expression recognition is the work of judging the type of facial expression in the picture through a certain program algorithm. The classification of facial expressions in the field is mainly based on the research of Paul Ekman, a famous American psychologist, and is divided into seven categories: surprise, sadness, anger, fear, happiness, disgust and contempt. Usually, facial expression recognition is divided into three stages: image preprocessing, facial expression feature extraction, and facial expression classification and recognition. Different stages have various forms and methods.

[0003] The main purpose of the image preprocessing stage is to increase and focus on key information of the image, reduce the impact of image quality on the recognition effect, reduce image noise, and improve the robustness of the recognition algorithm. Commonly used methods include: face detection, illumination normalization, posture normalization, occlusion removal, and data enhancement.

[0004] The expression feature extraction stage is the key to the recognition step. Its main purpose is to extract important information elements related to the expression from the preprocessing result object, and provide a decision basis for further expression classification and judgment. The types of features usually extracted are: geometric features, statistical features, frequency domain features, and motion features. Common feature extraction methods include: overall recognition method and local recognition method, deformation extraction method and motion extraction method, geometric feature method and appearance feature method, etc.

[0005] The expression classification and recognition stage is the final stage of classifying and distinguishing expressions, and it is also the most important stage in the entire expression recognition process. Based on the same feature extraction results, different classifiers will have different performances, and the errors in classification effects are also uneven. Common methods for expression classification include probability model classification, linear classifier, K-value mean classification, support vector machine, hidden Markov model, neural network classifier, etc.

[0006] In summary, with the development of artificial intelligence technology, expression recognition has been well supported by technology and has been successfully applied in many industries. As a result, more accurate requirements and demands for expression recognition have also emerged in various fields. Therefore, it is most urgent and important to continuously adopt new technologies and seek new breakthroughs to solve technical problems in expression recognition, especially to improve recognition effects. Summary of the invention

[0007] Purpose of the invention: In order to overcome the deficiencies in the prior art, a method for facial expression recognition based on multi-element fusion decision-making is provided, which has more accurate classification and better recognition effect.

[0008] Technical solution: To achieve the above purpose, the present invention provides a facial expression recognition method based on multi-element fusion decision-making, comprising the following steps:

[0009] S1: A face detection algorithm based on a lightweight attention mechanism is used to perform localization preprocessing on the input face image;

[0010] S2: Based on the pre-processing positioning, facial feature extraction is performed, and the feature extraction is optimized with a unique weight ratio and learning mechanism during the extraction process;

[0011] S3: Based on the extracted feature information and supplemented by multi-element fusion, the classification decision basis is output to provide more accurate basic elements for the classification model;

[0012] S4: Based on the feature information and multi-element fusion classification, the expression of the object to be tested is classified.

[0013] Furthermore, the step S1 specifically includes the following steps:

[0014] A1: Through the improved convolutional neural network model integrated with the attention mechanism, combined with residual information for pooling and ReLU activation processing, the target face is locked and recognized;

[0015] A2: Smooth excitation processing of segmented blocks during target recognition;

[0016] A3: Through the cyclic iteration of the sliding segmentation block, the locking preprocessing of the face image in the object to be tested is completed.

[0017] Furthermore, in step A2, the following calculation method is used when calculating the sliding displacement of the segmented block:

[0018]

[0019] Among them, B edge Indicates the boundary position coordinates of the segmented block, B center Represents the coordinates of the center position of the segmentation block, s represents the total number of samples contained in the data set used for training the network, l represents the number of categories contained in the data set used for training the network, and γ represents the algorithm model complexity and overfitting adjustment parameters.

[0020] Furthermore, the facial feature extraction method in step S2 is: using a facial feature extraction method based on multi-instance learning and attention mechanism, weighting is configured according to the degree of influence of different facial areas on expression recognition, and a feature vector is extracted.

[0021] Furthermore, the extraction of the feature vector in step S2 is defined as follows:

[0022] O n =M n ×Weight+Pv

[0023] Among them, O n represents the result of feature vector extraction, M n Represents the feature map of instance n, Weight represents the weight configuration, and Pv represents the parameter variable.

[0024] Furthermore, in step S2, the attention mechanism is used to learn the weights, and the attention superposition weights are calculated as follows:

[0025] W n =η(M n )

[0026] Among them, W n represents the weight of instance n, η represents the attention execution mechanism, and M n Represents the feature map of instance n.

[0027] Furthermore, in step S2, the extracted features are further optimized, and the strategy is: n '=W n ×(O n ).

[0028] Furthermore, the step S3 specifically includes the following steps:

[0029] B1: Introducing age, gender and skin color as weighted discriminant multi-factors to further supplement the feature information;

[0030] B2: Combine and classify the three types of multivariate factors;

[0031] B3: After weighted fusion, the classification of the features of the object to be tested is judged, which serves as the basis for the expression classification and judgment to produce multi-factor fusion decision-making of the face.

[0032] Furthermore, the expression of the discrimination process in step B3 is as follows:

[0033]

[0034] Among them, C represents the multi-factor discrimination classification result of the object to be tested, c represents the basic classification vector, s represents the feature vector extracted in the previous step, N represents the total number of trees in the decision forest, and f n Represents a tree node in a decision forest.

[0035] Furthermore, the step S4 specifically includes the following steps:

[0036] C1: A random forest decision tree model based on multi-factor fusion is introduced to obtain the probability distribution of expression classification results based on the fusion of multiple factors;

[0037] The calculation process is expressed as: C(c n ,s)=1 / N∑C(c n ,f n ×s), where c n Indicates the nth classification selected;

[0038] C2: Based on feature information and multi-factor fusion, the classifier produces more objective, efficient and accurate facial expression classification results. The classification process is:

[0039] The present invention provides a facial expression recognition method with multi-factor fusion decision-making, which aims to achieve a better expression recognition effect by integrating and jointly making decisions on multiple factors that affect facial expression recognition based on a random forest decision tree. The present invention introduces multi-instance learning and attention mechanism in the facial feature extraction stage, and weights are configured according to the degree of influence of different facial regions on expression recognition, and the weights are learned by the attention mechanism. Age, gender and skin color are introduced as weighted discriminant multi-factors in the facial multi-factor fusion decision stage, and the weighted fusion has an effect on expression classification and discrimination. A random forest decision tree model based on multi-factor fusion is introduced in the expression recognition stage, and a more objective, efficient and accurate expression recognition method is constructed based on the fusion of multi-factors.

[0040] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0041] 1. A face detection algorithm based on a lightweight attention mechanism is used to pre-process the input face image, which improves and solves the problem of improving the accuracy of face pre-positioning.

[0042] 2. Based on the pre-processing positioning, facial feature extraction is performed. During the extraction process, the feature extraction is optimized with a unique weight ratio and learning mechanism. Based on the feature information and supplemented by multi-dimensional fusion, the classification decision basis is output to provide more accurate basic elements for the classification model. The accuracy problem of feature extraction according to the relationship between actual features and their multi-dimensional causes is solved.

[0043] 3. Based on the feature information and multi-element fusion classification, the expression classification of the object to be tested is carried out. The accuracy of expression classification based on multi-element fusion is improved.

[0044] 4. Based on the method of the present invention, a facial expression recognition tool was developed, which realized the detection and recognition of expressions, and the visualization statistics of classification probability, and the effect of the method of the present invention was tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is the main framework flow chart of the method of the present invention;

[0046] Figure 2 Diagram of the user interface of the facial expression recognition tool developed for the facial expression recognition method based on multi-dimensional fusion decision-making. DETAILED DESCRIPTION

[0047] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0048] The present invention provides a facial expression recognition method based on multi-element fusion decision making, such as Figure 1 As shown, the following steps are included:

[0049] S1: A face detection algorithm based on a lightweight attention mechanism is used to perform localization preprocessing on the input face image;

[0050] Specifically, it includes the following steps S1-1 to S1-3:

[0051] S1-1: Through the improved convolutional neural network model integrated with the attention mechanism, the residual information is combined for pooling and ReLU activation processing to lock and identify the target face;

[0052] S1-2: During the target recognition process, the segmented blocks are subjected to smooth excitation processing, and the following calculation method is used to calculate the sliding displacement of the segmented blocks:

[0053]

[0054] Among them, B edge Indicates the boundary position coordinates of the segmented block, B center Represents the coordinates of the center position of the segmentation block, s represents the total number of samples contained in the data set used for training the network, l represents the number of categories contained in the data set used for training the network, and γ represents the algorithm model complexity and overfitting adjustment parameters.

[0055] S1-3: Through the cyclic iteration of the sliding segmentation block, the locking preprocessing of the face image in the object to be tested is completed.

[0056] S2: Based on the pre-processing positioning, facial feature extraction is performed, and the feature extraction is optimized with a unique weight ratio and learning mechanism during the extraction process;

[0057] Specifically, it includes the following steps S2-1 to S2-3:

[0058] S2-1: A facial feature extraction method based on multi-instance learning and attention mechanism is used to configure weights according to the degree of influence of different facial regions on expression recognition, and extract feature vectors.

[0059] The extraction of feature vectors is defined as follows:

[0060] O n =M n ×Weight+Pv

[0061] Among them, O n represents the result of feature vector extraction, M n Represents the feature map of instance n, Weight represents the weight configuration, and Pv represents the parameter variable.

[0062] S2-2: In order to avoid the influence of interference factors and improve the hit rate of feature extraction, the attention mechanism is used to learn the weights. The calculation method of attention superposition weights is:

[0063] W n =η(M n )

[0064] Among them, W n represents the weight of instance n, η represents the attention execution mechanism, and M n Represents the feature map of instance n.

[0065] S2-3: Further optimize the extracted features. The strategy is: n '=W n ×(O n ).

[0066] S3: Based on the extracted feature information and supplemented by multi-element fusion, the classification decision basis is output to provide more accurate basic elements for the classification model;

[0067] Specifically, it includes the following steps S3-1 to S3-3:

[0068] S3-1: Introduce age, gender and skin color as weighted discriminant multi-factors to further supplement the feature information;

[0069] It should be noted that the expression analysis for minors (0-17 years old) is a special type of research, so this embodiment is only for adults over 18 years old. Based on the latest national age group classification, the age group is divided into three categories: 18-45 years old (A1), 46-69 years old (A2), and over 69 years old (A3); gender is divided into: female (F), male (M); skin color is divided into three categories: Caucasian (white series W), East Asian (yellow series Y) and Negro-Australian (black series B).

[0070] S3-2: Combine and classify the three types of multivariate factors;

[0071] In this embodiment, it can be divided into the following 18 categories

[0072] A1,F,W A1,F,Y A1,F,B A1,M,W A1,M,Y A1,M,B A2,F,W A2,F,Y A2,F,B A2,M,W A2,M,Y A2,M,B A3,F,W A3,F,Y A3,F,B A3,M,W A3,M,Y A3,M,B

[0073] S3-3: After weighted fusion, the classification to which the features of the object to be tested belong is judged, which serves as the basis for the expression classification and judgment to produce the multi-factor fusion decision-making effect of the face.

[0074] The expression of the discrimination process is as follows:

[0075]

[0076] Among them, C represents the multi-factor discrimination classification result of the object to be tested, c represents the basic classification vector, s represents the feature vector extracted in the previous step, N represents the total number of trees in the decision forest, and f n Represents a tree node in a decision forest.

[0077] S4: Based on the feature information and multi-element fusion classification, the expression of the object to be tested is classified.

[0078] Specifically, it includes the following steps S4-1 and S4-2:

[0079] S4-1: Introduce the random forest decision tree model based on multi-factor fusion, and rely on the fusion of multiple factors to obtain the probability distribution of expression classification results;

[0080] The calculation process is expressed as: C(c n ,s)=1 / N∑C(c n ,f n ×s), where c n Indicates the nth classification selected;

[0081] S4-2: Based on the feature information and multi-factor fusion, the classifier produces a more objective, efficient and accurate facial expression classification result. The classification process is as follows:

[0082]

[0083] This embodiment also provides a facial expression recognition system based on multi-element fusion decision-making, which includes a network interface, a memory and a processor; wherein the network interface is used to realize the reception and transmission of signals in the process of sending and receiving information between other external network elements; the memory is used to store computer program instructions that can be run on the processor; the processor is used to execute the steps of the above-mentioned consensus method when running the computer program instructions.

[0084] The present embodiment also provides a computer storage medium, which stores a computer program, and the method described above can be implemented when the processor executes the computer program. The computer readable medium can be considered to be tangible and non-temporary. Non-limiting examples of non-temporary tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives) and optical storage media (such as CDs, DVDs or Blu-ray discs), etc. The computer program includes processor executable instructions stored on at least one non-temporary tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, a device driver that interacts with a specific device of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] Based on the above scheme, in order to verify the effectiveness and practical effect of the method of the present invention, this embodiment applies the technical scheme of the above method of the present invention to develop a facial expression recognition tool based on a facial expression recognition method based on multi-element fusion decision-making. The recognition tool is used to recognize facial expressions. The specific recognition results are as follows: Figure 2 As shown, it can be seen that not only can the detection and recognition of facial expressions be realized, but also the visualization statistics of expression classification probability can be realized, thereby verifying the effectiveness and practical effect of the method of the present invention.

[0090] In order to more intuitively reflect the effect of the method of the present invention, the classification accuracy of the method of the present invention is compared with the classification accuracy of common expression classification methods, as shown in the following table.

[0091] Common expression classification methods Highest classification accuracy ProposedTFP 94.84 2B(N+M)Softmax 97.10 DBN 93.0 This method (see Figure 2 ) 98.59

[0092] It can be seen that the classification accuracy of the method of the present invention has been significantly improved, which further verifies the practical effect of the present invention.

Claims

1. A facial expression recognition method based on multivariate fusion decision-making, characterized in that: The steps include: S1: A face detection algorithm based on a lightweight attention mechanism is used to perform localization preprocessing on the input face image; S2: Based on the pre-processing positioning, facial feature extraction is performed, and the feature extraction is optimized with a unique weight ratio and learning mechanism during the extraction process; S3: Based on the extracted feature information and supplemented by multi-element fusion, output the basis for classification decision; S4: Based on the feature information and multi-element fusion classification, the expression classification of the object to be tested is performed; The facial feature extraction method in step S2 is: using a facial feature extraction method based on multi-instance learning and attention mechanism, weighting is configured according to the influence of different facial regions on expression recognition, and a feature vector is extracted; The extraction of the feature vector in step S2 is defined as follows: O n =M n ×Weight+Pv Among them, O n represents the result of feature vector extraction, M n represents the feature map of instance n, Weight represents the weight configuration, and Pv represents the parameter variable; In step S2, the attention mechanism is used to learn the weights, and the attention superposition weights are calculated as follows: W n =η(M n ) Among them, W n represents the weight of instance n, η represents the attention execution mechanism, and M n Represents the feature map of instance n; In step S2, the extracted features are further optimized, and the strategy is: n '=W n ×(O n ).

2. The method for facial expression recognition based on multi-element fusion decision-making according to claim 1, characterized in that: The step S1 specifically includes the following steps: A1: Through the improved convolutional neural network model integrated with the attention mechanism, combined with residual information for pooling and ReLU activation processing, the target face is locked and recognized; A2: Smooth excitation processing of segmented blocks during target recognition; A3: Through the cyclic iteration of the sliding segmentation block, the locking preprocessing of the face image in the object to be tested is completed.

3. The method for facial expression recognition based on multi-element fusion decision-making according to claim 2, characterized in that: In step A2, the following calculation method is used to calculate the sliding displacement of the segmented block: Among them, B edge Indicates the boundary position coordinates of the segmented block, B center Represents the coordinates of the center position of the segmentation block, s represents the total number of samples contained in the data set used for training the network, l represents the number of categories contained in the data set used for training the network, and γ represents the algorithm model complexity and overfitting adjustment parameters.

4. The method for facial expression recognition based on multi-element fusion decision-making according to claim 1, characterized in that: The step S3 specifically includes the following steps: B1: Introducing age, gender and skin color as weighted discriminant multi-factors to further supplement the feature information; B2: Combine and classify the three types of multivariate factors; B3: After weighted fusion, the classification of the features of the object to be tested is judged, which serves as the basis for the expression classification and judgment to produce multi-factor fusion decision-making of the face.

5. The method for facial expression recognition based on multivariate fusion decision-making according to claim 4 is characterized in that: The expression of the discrimination process in step B3 is as follows: Among them, C represents the multi-factor discrimination classification result of the object to be tested, c represents the basic classification vector, s represents the feature vector extracted in the previous step, N represents the total number of trees in the decision forest, and f n Represents a tree node in a decision forest.

6. The method for facial expression recognition based on multi-element fusion decision-making according to claim 1, characterized in that: The step S4 specifically includes the following steps: C1: A random forest decision tree model based on multi-factor fusion is introduced to obtain the probability distribution of expression classification results based on the fusion of multiple factors; The calculation process is expressed as: C(c n ,s)=1 / N∑C(c n ,f n ×s), where c n Indicates the nth classification selected; C2: The facial expression classification result generated by the classifier based on feature information and multi-factor fusion. The classification process is: