Facial expression classification method and system

Abstract

The invention belongs to the technical field of data representation and signal classification, and relates to a face expression classification method and system. The system comprises a facial bioelectric sensor, a hardware data noise filtering module and an upper computer, the facial bioelectric sensor is used for collecting facial myoelectricity, electro-oculogram signals and posture data, and the hardware data noise filtering module is used for processing the myoelectricity and electro-oculogram signals; the method comprises the following steps: performing feature extraction on myoelectricity and electro-oculogram data subjected to digital-to-analog conversion to obtain time domain, frequency domain and transform domain features, calculating correlation coefficients of the features obtained by feature extraction, and performing intersection and fusion after judgment; and calculating an average MIC, performing screening and feature alignment, and classifying obtained selected data to obtain expression types. According to the method and the system, the number and the type of significant features can be increased by performing feature crossing and feature fusion according to correlation coefficients and threshold judgment; compared with the existing method and system, the method and the system have the advantage that the recognition accuracy is higher.

Description

Technical Field

[0001] The invention belongs to the technical field of data representation and signal classification, and relates to a method and system for classifying facial expressions of human faces. Background Art

[0002] With the rapid development of virtual reality (VR) and augmented reality (AR), VR technology has been widely used in the rehabilitation of autistic patients, social skills training for children, cognitive training for the elderly, telephone conferencing, etc. Facial expression is one of the important ways for human emotional communication. The demand for expression recognition in the field of human-computer interaction is growing. Facial expression also plays a key role in VR technology. Therefore, facial expression recognition technology has become a hot topic in current research.

[0003] In facial expression VR recognition technology, traditional cameras use facial tracking to identify the facial expressions of VR users, because VR will block the camera due to wearing. In recent years, many scholars have tried to use wearable devices to collect facial bioelectric signals of users immersed in VR experience, such as E0G, FEMG and other biological signals, but due to the number of electrodes and algorithm reasons, the recognition accuracy is low, and during long-term wearing, medical electrode patches are used to collect physiological signals. Long-term wearing of medical electrodes can easily cause skin irritation and inflammation. The use of MXene-based hydrogel electrodes can solve the problem of wearing affecting VR use.

[0004] This application is dedicated to improving the low recognition accuracy and effectively alleviating the problem of long-term wearing of medical electrodes by using electromyographic and electrooculographic signals collected by hydrogel electrodes through signal preprocessing, data preprocessing, feature extraction, crossover, fusion, screening, statistics and machine learning classification. Summary of the invention

[0005] The purpose of the present invention is to propose a method and system for classifying facial expressions of human faces in view of the technical status that the existing methods and systems for classifying facial expressions of human faces have poor signal quality, poor feature extraction effect and poor facial expression classification performance. The system includes a facial bioelectric sensor, a hardware data noise filtering module and a host computer; the facial bioelectric sensor is used to collect facial electromyography, electrooculography signals and posture data, and the hardware data noise filtering module is used to process electromyography and electrooculography signals; the method extracts features from the electromyography and electrooculography data after digital-to-analog conversion to obtain time domain, frequency domain and transform domain features, calculates the correlation coefficient of the features extracted, and crosses and fuses them after judgment; then calculates the average MIC and performs screening and feature alignment, and the selected data is classified to obtain the expression type. The method and system judge according to the correlation coefficient and threshold and then perform feature crossover and feature fusion to increase the number and type of significant features; and then classify after statistics, screening and alignment, and the recognition accuracy is higher than that of the existing methods and systems.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In a first aspect, the present invention provides a facial expression classification system, comprising a data acquisition unit, a hardware data noise filtering module, a data preprocessing module and a machine learning expression classification module, wherein the data acquisition unit comprises a plurality of electrodes, the data acquisition unit is connected to the hardware data noise filtering module, the hardware data noise filtering module is connected to the data preprocessing module, the data preprocessing module output is used for feature processing and selection, and further comprises an electromyography feature processing module, an electrooculography feature processing module and a feature selection module; the data acquisition unit acquires and outputs facial electromyography signals and electrooculography signals; the facial electromyography signals and electrooculography signals enter the hardware data noise filtering module and the data preprocessing module respectively to obtain preprocessed electromyography and electrooculography data; the preprocessed electromyography and electrooculography data are respectively subjected to the electromyography feature processing module and the electrooculography feature processing module to extract features, calculate correlation coefficients and cross the preprocessed electromyography and electrooculography data that meet a threshold value to obtain feature-processed electromyography and electrooculography data; the feature-processed electromyography and electrooculography data are subjected to feature fusion, statistics, screening and alignment by the feature selection module to obtain feature-selected data; the feature-selected data are classified by the machine learning expression classification module to obtain expression types.

[0008] The electromyographic feature processing module includes an electromyographic feature extraction unit, and the electrooculographic feature processing module includes an electrooculographic feature extraction unit; the electromyographic feature extraction unit extracts the time domain features, frequency domain features, statistical features and transform domain features of the facial electromyographic data after data preprocessing to obtain electromyographic extraction features; the electrooculographic feature extraction unit extracts the time domain features, frequency domain features, statistical features and transform domain features of the electrooculographic data after data preprocessing to obtain electrooculographic extraction features.

[0009] The electromyographic feature processing module also includes an electromyographic correlation coefficient calculation unit, an electromyographic correlation coefficient judgment unit and an electromyographic feature crossover unit; the electromyographic correlation coefficient calculation unit calculates the electromyographic correlation coefficient of the electromyographic extraction feature; the electromyographic correlation coefficient judgment unit judges whether the calculated electromyographic correlation coefficient is less than the electromyographic correlation coefficient threshold value, if so, the electromyographic extraction feature corresponding to the electromyographic correlation coefficient is sent to the electromyographic feature crossover unit for crossover to obtain electromyographic data after feature processing; otherwise, if it is greater than or equal to the electromyographic correlation coefficient threshold value, the electromyographic extraction feature corresponding to the electromyographic correlation coefficient is sent to the feature selection module as the electromyographic data after feature processing.

[0010] The electrooculogram feature processing module also includes an electrooculogram correlation coefficient calculation unit, an electrooculogram correlation coefficient judgment unit and an electrooculogram feature crossing unit; the electrooculogram correlation coefficient calculation unit calculates the electrooculogram correlation coefficient of the electrooculogram extracted feature; the electrooculogram correlation coefficient judgment unit judges whether the calculated electrooculogram correlation coefficient is less than the electrooculogram correlation coefficient threshold value, if so, the electrooculogram extracted feature corresponding to the electrooculogram correlation coefficient is sent to the electrooculogram feature crossing unit for crossing to obtain the electrooculogram data after feature processing; if it is greater than or equal to the electrooculogram correlation coefficient threshold value, the electrooculogram extracted feature corresponding to the electrooculogram correlation coefficient is sent to the feature selection module as the electrooculogram data after feature processing.

[0011] In a specific implementation, the intersection of the electromyographic feature intersection unit and the electrooculographic feature intersection unit is realized by using the Kronecker product to expand the number of significant features;

[0012] The feature selection module includes a feature fusion unit, an average MIC calculation unit, a feature screening unit and a feature alignment unit; the feature fusion unit receives the feature-processed electromyographic data and electrooculographic data sent by the electromyographic feature processing module and the electrooculographic feature processing module respectively and fuses them to obtain fused features; the average MIC calculation unit calculates the average MIC of the fused features; the feature screening unit screens the fused features according to the calculated average MIC to obtain screened features; the feature alignment unit aligns the screened features to obtain aligned features; and then the aligned features are input into the machine learning expression classification module.

[0013] The fusion is one or more combinations of addition, concatenation, multi-level fusion and multi-scale fusion.

[0014] The fusion can derive new effective feature types;

[0015] When the screening is implemented, in addition to screening based on the calculated average MIC, statistical screening, model screening and embedded screening can also be used; the statistical screening includes one or more of variance selection, chi-square test, mutual information and recursive methods; the model screening includes one or a combination of Pandas module screening, Sklearn module screening, LSTM model screening, CNN model screening and DNN model screening; the embedded screening includes one or a combination of filtering, wrapping and tree types.

[0016] The classification system calculates the average MIC of the feature-processed electromyographic and electrooculographic data, and then screens and selects or combines other screening methods to obtain screened features, and then aligns the features to obtain feature-selected data. The feature-selected data not only increases the number of significant features, but also obtains new effective or significant feature types.

[0017] In a second aspect, the present invention provides a method for classifying facial expressions of human faces, comprising the following steps:

[0018] S1, collect and obtain facial electromyography signals and electrooculography signals;

[0019] S2, preprocessing facial electromyographic signals and electrooculographic signals respectively;

[0020] S3, performing analog-to-digital conversion on the facial electromyography signal and electrooculography signal after signal preprocessing respectively;

[0021] S4, preprocessing the facial electromyography data and electrooculography data obtained after analog-to-digital conversion;

[0022] S5, performing feature processing on the facial electromyography data and electrooculography data obtained after data preprocessing respectively;

[0023] S6, performing feature selection on the facial electromyography features and electrooculography features obtained after feature processing to obtain aligned features;

[0024] S7. Classify the aligned features obtained after feature selection through a machine learning model to obtain expression types.

[0025] In the specific implementation of S1, the facial electromyographic signal and the electrooculographic signal are collected and acquired by a data acquisition unit, and the acquisition unit collects and obtains bioelectric signals with high signal-to-noise ratio and high conductivity through a plurality of electrodes fixed inside the AR eye mask;

[0026] The bioelectric signals include facial muscle electrical signals and eye electrical signals;

[0027] The high signal-to-noise ratio is greater than or equal to 25 dB; the high conductivity is greater than or equal to 0.8×10 -2 S cm -1 The electrode is a MXene-based hydrogel electrode, which is prepared by MXene-based hydrogel, and the preparation process has the advantages of simple process, uniform MXene distribution and biocompatibility;

[0028] The S2 performs signal preprocessing on the facial electromyographic signal, specifically: performing analog low-pass filtering on the electromyographic signal; performing a first-stage amplification on the electromyographic signal after the analog low-pass filtering; performing analog high-pass filtering on the electromyographic signal after the first-stage amplification; performing a second-stage amplification on the electromyographic signal after the analog high-pass filtering to obtain the facial electromyographic signal after signal preprocessing; performing signal preprocessing on the electrooculogram signal, specifically: performing analog low-pass filtering on the electrooculogram signal; performing a first-stage amplification on the electrooculogram signal after the analog low-pass filtering; performing analog high-pass filtering on the electrooculogram signal after the first-stage amplification; performing a second-stage amplification on the electrooculogram signal after the analog high-pass filtering to obtain the electrooculogram signal after signal preprocessing;

[0029] S4 performs data preprocessing on the facial electromyography data obtained after analog-to-digital conversion, specifically: digitally filtering the electromyography data after analog-to-digital conversion, normalizing the electromyography data after digital filtering, and obtaining the electromyography data after data preprocessing; performs data preprocessing on the electrooculography data obtained after analog-to-digital conversion, specifically: digitally filtering the electrooculography data after analog-to-digital conversion, normalizing the electrooculography data after digital filtering, and obtaining the electrooculography data after data preprocessing.

[0030] The S5 performs feature processing on the facial electromyography data obtained after data preprocessing, specifically: extracting the time domain, frequency domain, statistical and transform domain features of the facial electromyography data after data preprocessing to obtain electromyography extraction features; calculating the electromyography correlation coefficient of the electromyography extraction features, judging whether the calculated electromyography correlation coefficient is less than the electromyography correlation coefficient threshold, if it is less than the electromyography correlation coefficient threshold, then crossing the electromyography extraction features corresponding to the electromyography correlation coefficient to obtain the electromyography data after feature processing; otherwise, if it is greater than or equal to the electromyography correlation coefficient threshold, then using the electromyography extraction features corresponding to the electromyography correlation coefficient as the electromyography data after feature processing;

[0031] The electrooculogram data obtained after data preprocessing are subjected to feature processing, specifically: time domain, frequency domain, statistical and transform domain features of the electrooculogram data after data preprocessing are extracted to obtain electrooculogram extraction features; the electrooculogram correlation coefficient of the electrooculogram extraction features is calculated; it is determined whether the correlation coefficient calculated by the electrooculogram correlation coefficient calculation unit is less than the electrooculogram correlation coefficient threshold value, if it is less than the electrooculogram correlation coefficient threshold value, the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient are crossed to obtain the electrooculogram data after feature processing; otherwise, if it is greater than or equal to the electrooculogram correlation coefficient threshold value, the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient are used as the electrooculogram data after feature processing.

[0032] The S6, feature selection of the facial electromyographic features and electrooculographic features obtained after feature processing, specifically: respectively receiving the electromyographic and electrooculographic data after feature processing and fusing them to obtain fused features; calculating the average MIC of the fused features; selecting the first N fused features corresponding to the calculated average MIC value as the screened features; aligning the screened features to obtain aligned features; the fusion includes one or more combinations of addition, series connection, multi-level fusion and multi-scale fusion.

[0033] The fusion can derive new effective feature types; when the screening is implemented, in addition to screening based on the calculated average MIC, statistical screening, model screening and embedded screening can also be used; the statistical screening includes one or more of variance selection, chi-square test, mutual information and recursive methods; the model screening includes one or a combination of Pandas module screening, Sklearn module screening, LSTM model screening, CNN model screening and DNN model screening; the embedded screening includes one or a combination of filtering, wrapping and tree types.

[0034] In general, the innovations of the classification method and system include two aspects:

[0035] On the one hand, acquisition, signal preprocessing and data preprocessing;

[0036] The acquisition is achieved by a data acquisition unit with a plurality of electrodes, wherein the electrodes are MXene-based hydrogel electrodes, which have a simple preparation process, uniform distribution of MXene on the electrodes and good biocompatibility; the collected facial muscle electrical signals and eye electrical signals have the advantages of high signal-to-noise ratio and high conductivity;

[0037] The signal preprocessing is to perform cross-secondary filtering and secondary amplification on the collected electromyographic signals and electrooculographic signals, namely: low-pass filtering, primary amplification, high-pass filtering, and secondary amplification; in specific implementation: the low-pass filtering and high-pass filtering of the electromyographic signals can be exchanged, and the primary amplification and secondary amplification factors are adjusted according to the signal quality before and after processing; the amplification factors of the primary amplification and the secondary amplification are greater than or equal to 10 and less than or equal to 200; the cut-off frequency of the low-pass filtering is greater than or equal to 10 Hz and less than or equal to 50 Hz; the cut-off frequency of the high-pass filtering is greater than or equal to 200 Hz and less than or equal to 600 Hz;

[0038] In specific implementation: the low-pass filtering and high-pass filtering of the electrooculogram signal can also be exchanged in order, and the primary amplification and secondary amplification factors are adjusted according to the signal quality before and after processing; the amplification factors of the primary amplification and secondary amplification are greater than or equal to 10 and less than or equal to 200; the cutoff frequency of the low-pass filtering is less than or equal to 20Hz; the cutoff frequency of the high-pass filtering is greater than or equal to 20Hz and less than or equal to 100Hz;

[0039] When the data preprocessing is specifically implemented, in addition to digital filtering and normalization, a data scaling unit is added before or after digital filtering to adjust the numerical range; a data enhancement unit can also be added before normalization to adjust the numerical range while highlighting data features to facilitate normalization processing.

[0040] On the other hand, the innovation lies in the feature processing and feature selection part;

[0041] The feature processing is specifically implemented by extracting the time domain, frequency domain, statistical and transform domain features of the facial electromyography data after data preprocessing to obtain electromyography extraction features; in specific implementation, decomposition features can also be used;

[0042] The time domain characteristics include, but are not limited to, electromyographic integral value, root mean square value, waveform length and average absolute value;

[0043] The frequency domain features include but are not limited to features after FFT and STFT transformation;

[0044] The statistical characteristics include, but are not limited to, variance and slope characteristics;

[0045] The transform domain features include, but are not limited to, CWD distribution features;

[0046] The decomposition features include, but are not limited to, SVD decomposition;

[0047] The feature selection is specifically implemented as follows: receiving the feature-processed electromyographic and electrooculographic data respectively and fusing them to obtain fused features; calculating the average MIC of the fused features; selecting the first N fused features corresponding to the calculated average MIC value as the screened features; aligning the screened features to obtain aligned features;

[0048] The fusion includes one or more combinations of addition, concatenation, multi-level fusion and multi-scale fusion; the fusion can derive new effective feature types;

[0049] When the screening is implemented, in addition to screening based on the calculated average MIC, statistical screening, model screening and embedded screening can also be used; the statistical screening includes but is not limited to variance selection, chi-square test, mutual information and recursive method; the model screening includes one or a combination of Pandas module screening, Sklearn module screening, LSTM model screening, CNN model screening and DNN model screening; the embedded screening includes one or a combination of filtering, wrapping and tree types.

[0050] When the present invention is implemented, the aligned features are outputted by relying on the aforementioned feature processing and feature selection, which significantly increases the number and types of effective features, and can obtain a more accurate recognition rate compared to existing methods and systems.

[0051] Beneficial Effects

[0052] Compared with the existing facial expression classification methods and systems, the present invention provides a method and system for classifying facial expressions of human faces, which has the following beneficial effects:

[0053] 1. The classification system, relying on the host computer to extract time domain, frequency domain and transform domain features from the EMG and EOG data after digital-to-analog conversion, calculate the correlation coefficient between the features, select the features that meet the threshold, and then cross them, using the Kronecker product; the number of significant features of the feature expansion obtained;

[0054] 2. The classification method and system calculate the average MIC of the feature-processed electromyographic and electrooculographic data, and then perform screening and feature alignment to obtain the feature-selected data, which can increase the number of significant features and fuse effective feature types;

[0055] 3. The classification method and system rely on the significant and effective features of increasing the number and types of feature processing and feature selection outputs, and can obtain a more accurate recognition rate compared to existing methods and systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of the composition and connection of a human face facial expression classification system of the present invention;

[0057] Figure 2 A flow chart of a method for classifying facial expressions of a human face according to the present invention;

[0058] Figure 3 A schematic diagram of electrode distribution when a human face facial expression classification system is implemented according to the present invention;

[0059] Figure 4 This is a diagram showing the SVM classification effect in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The specific implementation of a method and system for classifying facial expressions of a human face proposed by the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0061] Example 1

[0062] The present invention proposes a method and system for classifying facial expressions of human faces, which can be applied to VR technology; human-computer interaction, such as allowing robots to better judge human facial expression features and thus better judge human emotions. The system includes a facial bioelectric sensor, a hardware data noise filtering module and a host computer; the facial bioelectric sensor is used to collect facial electromyography, electrooculography signals and posture data, and the hardware data noise filtering module is used to process electromyography and electrooculography signals; the method extracts features from the electromyography and electrooculography data after digital-to-analog conversion to obtain time domain, frequency domain and transform domain features, calculates the correlation coefficient of the features extracted, and crosses and fuses them after judgment; then calculates the average MIC and performs screening and feature alignment, and classifies the selected data to obtain the expression type. The method and system can increase the number and type of significant features by performing feature crossover and feature fusion based on correlation coefficient and threshold judgment; and then classifies after statistics, screening and alignment, which has a higher recognition accuracy than existing methods and systems.

[0063] Figure 1 The present invention is a composition and connection diagram of a facial expression classification system of a human face; the classification system includes a signal acquisition unit, a signal preprocessing module (which realizes hardware noise filtering and multi-stage amplification of each signal from a hardware perspective), a data preprocessing module, a feature processing module and a machine learning expression classification module, the signal acquisition unit includes a plurality of electrodes, the unit is connected to the signal preprocessing module, the signal preprocessing module is connected to the data preprocessing module, the data preprocessing module is connected to the feature processing module, and the feature processing module is connected to the machine learning expression classification module; the feature processing module includes an electromyography feature processing module and an electrooculography feature processing module; the data preprocessing module outputs preprocessed data mainly subjected to digital filtering and normalization for feature processing and selection;

[0064] When normalization is specifically implemented, it includes row normalization and column normalization based on various norms, such as the 2 norm;

[0065] After data preprocessing, the EMG and EOG data are respectively processed by the EMG feature processing module and the EOG feature processing module to extract features, calculate correlation coefficients, and cross the preprocessed EMG and EOG data that meet the threshold to obtain the EMG and EOG data after feature processing; the EMG and EOG data after feature processing are then processed by the feature selection module for feature fusion, statistics, screening and alignment to obtain the data after feature selection; the data after feature selection is classified by the machine learning expression classification module to obtain the expression type.

[0066] In a specific implementation, the data acquisition unit includes a plurality of electrodes, wherein the electrodes are prepared by MXene-based hydrogel, and the preparation process has the advantages of simple process, uniform MXene distribution and biocompatibility; the bioelectric signals (including facial muscle electrical signals and eye electrical signals) collected by the MXene-based hydrogel electrodes have high signal-to-noise ratio and high conductivity;

[0067] In specific implementation, the signal quality of facial muscle electrical signals and eye electrocardiogram signals collected by MXene-based hydrogel electrodes is better than that of signals collected by existing medical electrodes; the average signal-to-noise ratio of bioelectric signals collected by medical electrodes is less than 24dB; the conductivity does not exceed 0.8×10 -2 S cm -1 In this embodiment, the average signal-to-noise ratio of the facial muscle electrical signals and eye electrogram signals collected by the MXene-based hydrogel electrodes is not less than 26 dB; the conductivity is greater than 1×10 -2 S cm -1 The acquisition of signals with high signal-to-noise ratio and high conductivity is more conducive to later data processing and classification, and improves classification accuracy.

[0068] In the specific implementation, test data of 8 subjects are collected, and the 8 subjects are 4 males and 4 females. The subjects wear multiple electrodes distributed on the face, and the multiple electrodes include 4 surface muscle electrodes and 4 eye movement signal collection electrodes.

[0069] Electrode distribution of prior art: In the prior art, there are many collecting electrodes, which are usually distributed at the positions of seven muscles commonly used in facial expressions. Taking the middle of the forehead as the reference electrode, the electrodes are distributed at the positions of the frontalis muscle, corrugator supercilii muscle, orbicularis oculi muscle, upper lip and levator alar nasi muscle, zygomaticus muscle, orbicularis oris muscle, and depressor anguli oris muscle. Due to the symmetry of facial muscles, seven pairs of electrodes are required, which is a large number of electrodes.

[0070] The traditional layout for collecting electrooculogram signals is that the electrodes are placed above, below, left, and right of the eyes, with the reference electrode in the middle of the forehead;

[0071] The electrode distribution of this application: combining the characteristics of electrooculogram and electromyography, the electrooculogram electrodes are distributed on the left and right sides of both eyes and the upper and lower sides of one eye to collect electrooculogram and electromyography. There are four electrodes in total, distributed in the levator labii superioris and palpebral muscles. The existing electrode layout only places four electrodes on the forehead without a reference electrode. The electrode layout of this application is different from the existing electrode layout: for EOG signals, the traditional layout is that the electrode patches are arranged on the upper, lower, left and right sides of the eyes, and there is another electrode as a reference electrode.

[0072] In order to collect facial measurement data using a hardware acquisition system, the hardware system includes a facial bioelectric sensor, a hardware data noise filtering module and a host computer;

[0073] The facial bioelectric sensor is a data acquisition unit, including an electromyographic signal acquisition unit and an electrooculographic signal acquisition unit;

[0074] The facial bioelectric sensor includes eight MXene-based hydrogel electrodes; the facial measurement data includes muscle electrical signals, inertial sensor signals and eye electrical signals, the surface muscle electrical signals are facial muscle electrical signals and are collected by a muscle electrical signal collection unit; the eye electrical signals are collected by an eye electrical signal collection unit;

[0075] The electrode distribution of the electrooculogram signal acquisition unit and the facial muscle signal acquisition unit is as follows:

[0076] The hardware data noise filtering module includes a facial surface muscle electroelectric module, an electrooculogram module, and a mainboard; the facial muscle electroelectric signal module processes facial muscle electroelectric signals; the electrooculogram module is used to process electrooculogram signals, and the mainboard is used for analog-to-digital conversion and power supply;

[0077] The signal preprocessing module, when implemented specifically, realizes hardware noise filtering and multi-stage amplification of each signal from a hardware perspective, including an electromyographic signal preprocessing module and an electrooculographic signal preprocessing module; the electromyographic signal preprocessing module includes a low-pass filtering circuit, a primary amplifying circuit, a high-pass filtering circuit and a secondary amplifying circuit; after the facial muscle electrical signal is collected, the low-pass filtering circuit realizes preliminary denoising of the filtering, filtering out noise above 500Hz; the primary amplifying circuit realizes the gain of the circuit; the high-pass filtering circuit realizes further denoising of the filtering, filtering out noise below 10Hz; the secondary amplifying circuit realizes the final gain of the circuit; the primary amplifying circuit and the secondary amplifying circuit of the electromyographic signal, when implemented specifically, amplify by 15 times and 20 times respectively;

[0078] The electrooculogram signal preprocessing module completes the preprocessing of the electrooculogram signal, and also includes a low-pass filter circuit, a first-level amplifier circuit, a high-pass filter circuit and a second-level amplifier circuit; after the electrooculogram signal is collected, due to the difference between the electrooculogram signal and the facial muscle signal, the low-pass filter circuit realizes preliminary denoising of the filter and filters out noise above 38Hz; the first-level amplifier circuit realizes the gain of the circuit; the high-pass filter circuit realizes further denoising of the filter and filters out noise below 0.50Hz; the second-level amplifier circuit realizes the final gain of the circuit; the first-level amplifier circuit and the second-level amplifier circuit of the electrooculogram signal, when implemented, amplify by 20 times and 30 times respectively;

[0079] The measured electromyographic and electrooculographic signals are collected by the signal preprocessing module, converted into analogues, and then enter the data preprocessing module;

[0080] The data preprocessing includes digital filtering and normalization to obtain normalized facial electromyography data, including low-pass filtering of the preprocessed facial electromyography data converted into digital signals, and high-pass filtering of the preprocessed electrooculography data converted into digital signals; normalizing the filtered facial electromyography data and electrooculography data respectively to obtain normalized facial electromyography data and electrooculography data.

[0081] The normalized facial electromyography data and the normalized electrooculography data; the normalization is specifically implemented by using 2-norm normalization;

[0082] Perform feature extraction to extract the features of electrooculogram signals and facial electromyography signals respectively;

[0083] The facial electromyography signal features include but are not limited to time domain, frequency domain and transform domain features; the time domain features include common surface facial muscle electrical time domain features, such as: electromyography integral value, root mean square value, waveform length and average absolute value;

[0084] The time domain features of the electrooculogram signal include but are not limited to the difference between the peak and the trough value, the energy integral value, and the peak and trough position difference; in specific implementation, frequency domain features, transform domain features, statistical features and entropy features can also be extracted; the frequency domain features include FFT and STFT features; the transform domain features include wavelet features; the statistical features include but are not limited to mean, variance and high-order statistical features;

[0085] The extracted multiple electrooculogram signal features enter the electrooculogram correlation coefficient calculation unit to calculate the electrooculogram correlation coefficient, and then enter the electrooculogram correlation coefficient judgment unit to judge the calculated electrooculogram correlation coefficient. When the electrooculogram correlation coefficient is less than the threshold, the feature is cross-featured. In specific implementation, the electrooculogram correlation coefficient threshold is 0.5, that is, when the correlation is small, it indicates that this feature is very important and needs to enter the electrooculogram feature cross-feature unit for feature expansion. One of the feature expansion methods is: perform Kronecker product on the feature with itself; or perform nonlinear transformation on the feature first, such as tanh transformation, and then perform Kronecker product calculation on the feature with itself.

[0086] Similarly, extract multiple facial muscle electrical features and enter the electromyographic correlation coefficient calculation unit to calculate the electromyographic correlation coefficient, and then enter the electromyographic correlation coefficient judgment unit to judge the calculated electromyographic correlation coefficient. When the electromyographic correlation coefficient is less than the threshold, the feature is cross-featured. In specific implementation, the electromyographic correlation coefficient threshold is 0.45, that is, when the correlation is small, it indicates that this feature is very important and needs to enter the electromyographic feature cross-feature unit for feature expansion. One of the feature expansion methods is: perform Kronecker product on the feature with itself; or perform nonlinear transformation on the feature first, such as tanh transformation, and then perform Kronecker product calculation on the feature with itself.

[0087] The features output by the electromyographic feature cross unit and the electrooculographic feature cross unit enter the feature fusion unit for feature fusion, wherein the feature fusion includes but is not limited to addition fusion, product fusion, and fusion based on the self-attention mechanism;

[0088] The features obtained after fusion are first calculated by the average MIC calculation unit to calculate the average MIC, then input into the feature screening unit to screen the features, and finally enter into the feature alignment unit to obtain the aligned features for machine learning classification.

[0089] The feature screening, when implemented specifically, includes but is not limited to feature selection based on NCA and feature selection based on mRMR;

[0090] The feature alignment unit performs feature complementation on features of different dimensions and forms a set of vectors, which enter the machine learning expression classification module;

[0091] The machine learning expression classification module uses a commonly used machine learning classification algorithm such as a support vector machine (SVM) to effectively classify facial expressions;

[0092] The specific classification steps of the SVM vector machine are to map the data into a high-dimensional space, find the optimal hyperplane, and achieve classification and regression; the optimal hyperplane can maximize the maximum interval between different categories, so that the projections of the data points thereon are as separated as possible.

[0093] The test data of 8 subjects were collected, including 4 males and 4 females. The subjects wore multiple electrodes distributed on their faces. Figure 3 The electrode distribution of this embodiment: multiple electrodes include 4 surface muscle electrodes and 4 eye movement signal collection electrodes; the electrooculogram electrodes are distributed as ①, ②, ③, and ④; the myoelectric electrodes are distributed as ⑤, ⑥, ⑦, and ⑧, and ⑨ is the reference ground; the wearing position is around the eyes. The test data of the subjects under various expressions are collected. In the specific implementation, first, 3 groups of lip movements and 5 groups of eye movements of the subjects are collected, and then the next facial expression is shown to the participants, and the given facial expression is repeated 4 times; each collection time is 7s, starting from the 2nd second, continuing for 3s, and ending at the 5th second, with a rest time of 2 seconds.

[0094] like Figure 2 As shown, a flow chart of a method for classifying facial expressions of human faces is provided, comprising the following steps:

[0095] S1, collect facial electromyography and electrooculography signals;

[0096] S1 collects facial electromyographic and electrooculographic signals using multiple electrodes distributed on the face;

[0097] In a specific implementation, the data acquisition device for collecting facial electromyographic signals and electrooculographic signals also includes a wireless transmission unit for transmitting facial expression measurement data to a data processing terminal, such as a PC terminal or a remote server connected via a network. The wireless transmission method includes a mobile communication network, a WIFI network or Bluetooth.

[0098] S2, preprocessing facial electromyographic signals and electrooculographic signals respectively;

[0099] The step of preprocessing the electromyographic signal in S2 includes the following sub-steps:

[0100] S21A performs analog low-pass filtering on the EMG;

[0101] S21B performs the first-stage amplification of the EMG signal after analog low-pass filtering;

[0102] S21C performs analog high-pass filtering on the EMG after the first stage of amplification;

[0103] S21D performs the second stage amplification on the EMG signal after analog high-pass filtering;

[0104] The step of preprocessing the electrooculogram signal in S2 includes the following sub-steps:

[0105] S22A performs analog low-pass filtering on the electrooculogram;

[0106] S22B performs the first stage amplification of the EOG signal after analog low-pass filtering;

[0107] S22C performs analog high-pass filtering on the electrooculogram after the first stage of amplification;

[0108] S22D performs the second-stage amplification on the EOG signal after analog high-pass filtering;

[0109] In specific implementation, the low-pass filtering and high-pass filtering of S21B and S21D as well as S22B and S22D can be exchanged; the primary amplification and secondary amplification factors of S21A and S21C as well as S22A and S22C are adjusted according to the signal quality before and after processing; the amplification factors of the primary amplification and the secondary amplification are greater than or equal to 10 and less than or equal to 200;

[0110] In S2, the facial electromyography or electrooculography signals are amplified and filtered respectively, specifically including: the facial muscle electromyography signals pass through a low-pass filtering circuit, the low-pass filtering circuit realizes preliminary denoising of the filtering, and filters out noise above 500 Hz; the first-stage amplification circuit realizes the gain of the circuit; the high-pass filtering circuit realizes further denoising of the filtering, and filters out noise below 10 Hz; the second-stage amplification circuit realizes the final gain of the circuit.

[0111] S3, performing analog-to-digital conversion on the facial electromyography signal and electrooculography signal after signal preprocessing respectively;

[0112] S4, preprocessing the facial electromyography data and electrooculography data obtained after analog-to-digital conversion;

[0113] In S4, the facial electromyography data obtained after the analog-to-digital conversion is digitally filtered and normalized to obtain normalized facial electromyography data, specifically including: performing high-pass filtering on the facial electromyography data after baseline removal; performing low-pass filtering on the electrooculography data; in specific implementation, removing the baseline of the facial electromyography data (i.e., subtracting the mean) to obtain the facial electromyography data after baseline removal, and then performing high-pass filtering to filter out low-frequency noise at the digital frequency corresponding to 10 Hz;

[0114] S5, performing feature processing on the facial electromyography data and electrooculography data obtained after data preprocessing respectively;

[0115] The S5 performs feature processing on the facial electromyography data obtained after data preprocessing, specifically: extracting the time domain, frequency domain, statistical and transform domain features of the facial electromyography data after data preprocessing to obtain electromyography extraction features; calculating the electromyography correlation coefficient of the electromyography extraction feature, judging whether the calculated electromyography correlation coefficient is less than the electromyography correlation coefficient threshold, if it is less than the electromyography correlation coefficient threshold, then crossing the electromyography extraction features corresponding to the electromyography correlation coefficient to obtain the electromyography data after feature processing; if it is greater than or equal to the electromyography correlation coefficient threshold, then using the electromyography extraction features corresponding to the electromyography correlation coefficient as the electromyography data after feature processing;

[0116] The electromyography extraction feature is specifically implemented by respectively extracting the features of four different channels of facial muscle electroencephalogram data and four channels of electrooculogram data; according to the different time domain features of different eye movements and the difference in the amplitude of electrooculogram, the difference between the peak and trough values ​​of CH1 and CH2 and the energy integral value of the signal are used as classification features; the common time domain features of the surface muscle electroencephalogram signals of CH3 and CH4 are extracted; the common time domain features include electromyography integral value, root mean square value, waveform length and average absolute value.

[0117] The electrooculogram data obtained after data preprocessing are subjected to feature processing, specifically: time domain, frequency domain, statistical and transform domain features of the electrooculogram data after data preprocessing are extracted to obtain electrooculogram extraction features; the electrooculogram correlation coefficient of the electrooculogram extraction features is calculated; it is determined whether the correlation coefficient calculated by the electrooculogram correlation coefficient calculation unit is less than the electrooculogram correlation coefficient threshold value, if it is less than the electrooculogram correlation coefficient threshold value, the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient are crossed to obtain the electrooculogram data after feature processing; otherwise, if it is greater than or equal to the electrooculogram correlation coefficient threshold value, the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient are used as the electrooculogram data after feature processing.

[0118] S6, performing feature selection on the facial electromyography features and electrooculography features obtained after feature processing to obtain aligned features;

[0119] The S6 performs feature selection on the facial electromyographic features and electrooculographic features obtained after feature processing, specifically: respectively receiving the electromyographic and electrooculographic data after feature processing and fusing them to obtain fused features; calculating the average MIC of the fused features; selecting the first N fused features corresponding to the calculated average MIC value as the screened features; aligning the screened features to obtain aligned features; the fusion includes one or more combinations of addition, concatenation, multi-level fusion and multi-scale fusion.

[0120] The fusion can derive new effective feature types; when the screening is specifically implemented, in addition to screening based on the average MIC calculated in this embodiment, statistical screening, model screening and embedded screening can also be used; the statistical screening includes one or more of variance selection, chi-square test, mutual information and recursive methods; the model screening includes one or a combination of Pandas module screening, Sklearn module screening, LSTM model screening, CNN model screening and DNN model screening; the embedded screening includes one or a combination of filtering, wrapping and tree types.

[0121] S7. Classify the aligned features obtained after feature selection through a machine learning model to obtain expression types.

[0122] In step S7, the specific classification step of the SVM vector machine is to map the data into a high-dimensional space, find the optimal hyperplane, and achieve classification and regression; the optimal hyperplane can maximize the maximum interval between different categories, so that the projections of the data points thereon are as separated as possible;

[0123] S2, S3 and S4 respectively perform signal preprocessing, analog-to-digital conversion and data preprocessing on the collected signals. Among them, signal preprocessing includes double filtering and amplification, using a low-pass filter unit to filter out noise above 500Hz; and filtering out noise below 10Hz with a high-pass filter. After the filtered surface muscle electrogram signal and electrooculogram signal are converted into analog-to-digital, they are normalized to obtain normalized facial muscle electrogram and electrooculogram data. Next, a variety of extraction methods are used to extract features from the preprocessed facial surface muscle electrogram and electrooculogram data.

[0124] S5 extracts the time domain features of the electrooculogram signal, wherein the time domain features include the difference between the peak and the trough value, the energy integral value, and the peak and trough position difference. The facial electromyography signal features are extracted as the surface muscle electrical time domain features, including the electromyography integral value, the root mean square value, the waveform length, and the average absolute value;

[0125] The extracted features will form a one-dimensional vector with a length of fourteen, and the combination method is: Among them, vd is the peak-to-trough difference, ld is the peak-to-trough position difference, i is the energy integral, mav is the mean value, wl is the wavelength, rms is the root mean square, and iemg is the integrated value of fEMG. 1, 2, 3, and 4 are the serial numbers of the four channels respectively.

[0126] The features with 14 dimensions extracted above were used to collect 256 samples for 8 expressions. After the data were processed, fused and selected as in S5 and S6, the data of the last action of the subject was obtained and processed and selected as the test sample. The data collected from the first three actions were processed and selected as training samples for the training of four types of algorithms. The four types of algorithms are the machine learning classification algorithms Support Vector Machine (SVM), Random Forest (RF), K-Nearest Neighbor (KNN) and Linear Discriminant Analysis (LDA). The classification results are as follows: Figure 4 described.

[0127] Facial expressions are classified by machine learning algorithms to evaluate the performance of the collected data for expression classification. The obtained performance is shown in Table 1:

[0128] Table 1 Test results of different classification algorithms

[0129]

[0130] The accuracy is the accuracy of the classification between the model output result and the actual result. As shown in Table 1, the SVM accuracy is 95.3%. This model has the best effect in this embodiment.

[0131] The above is only a preferred embodiment of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A facial expression classification system, comprising a data acquisition unit, a hardware data noise filtering module, a data preprocessing module and a machine learning expression classification module, wherein the data acquisition unit comprises a plurality of electrodes, the data acquisition unit is connected to the hardware data noise filtering module, the hardware data noise filtering module is connected to the data preprocessing module, and the output of the data preprocessing module is used for feature processing and selection, characterized in that: It also includes an electromyographic feature processing module, an electrooculographic feature processing module and a feature selection module; the data acquisition unit collects and outputs facial electromyographic signals and electrooculographic signals; the facial electromyographic signals and electrooculographic signals enter the hardware data noise filtering module and the data preprocessing module respectively to obtain preprocessed electromyographic and electrooculographic data; The pre-processed electromyographic and electrooculographic data are respectively subjected to the electromyographic feature processing module and the electrooculographic feature processing module to extract features, calculate correlation coefficients, and cross the pre-processed electromyographic and electrooculographic data that meet the threshold to obtain the electromyographic and electrooculographic data after feature processing; the electromyographic and electrooculographic data after feature processing are then subjected to feature fusion, statistics, screening and alignment by the feature selection module to obtain the data after feature selection; After feature selection, the data is classified into expression types through the machine learning expression classification module.

2. A facial expression classification system according to claim 1, characterized in that: The electromyographic feature processing module includes an electromyographic feature extraction unit, and the electrooculographic feature processing module includes an electrooculographic feature extraction unit; the electromyographic feature extraction unit extracts the time domain features, frequency domain features, statistical features and transform domain features of the facial electromyographic data after data preprocessing to obtain electromyographic extraction features; The electrooculogram feature extraction unit extracts the time domain features, frequency domain features, statistical features and transform domain features of the electrooculogram data after data preprocessing to obtain electrooculogram extraction features.

3. A facial expression classification system according to claim 2, characterized in that: The electromyographic feature processing module also includes an electromyographic correlation coefficient calculation unit, an electromyographic correlation coefficient judgment unit and an electromyographic feature crossover unit; the electromyographic correlation coefficient calculation unit calculates the electromyographic correlation coefficient of the electromyographic extraction feature; the electromyographic correlation coefficient judgment unit judges whether the calculated electromyographic correlation coefficient is less than the electromyographic correlation coefficient threshold value, if so, the electromyographic extraction feature corresponding to the electromyographic correlation coefficient is sent to the electromyographic feature crossover unit for crossover to obtain electromyographic data after feature processing; if it is greater than or equal to the electromyographic correlation coefficient threshold value, the electromyographic extraction feature corresponding to the electromyographic correlation coefficient is sent to the feature selection module as the electromyographic data after feature processing.

4. A facial expression classification system according to claim 2, characterized in that: The electrooculogram feature processing module further includes an electrooculogram correlation coefficient calculation unit, an electrooculogram correlation coefficient judgment unit and an electrooculogram feature intersection unit; The electrooculogram correlation coefficient calculation unit calculates the electrooculogram correlation coefficient of the electrooculogram extraction feature; the electrooculogram correlation coefficient judgment unit judges whether the calculated electrooculogram correlation coefficient is less than the electrooculogram correlation coefficient threshold value, and if so, the electrooculogram extraction feature corresponding to the electrooculogram correlation coefficient is sent to the electrooculogram feature crossing unit for crossing to obtain the electrooculogram data after feature processing; if it is greater than or equal to the electrooculogram correlation coefficient threshold value, the electrooculogram extraction feature corresponding to the electrooculogram correlation coefficient is sent to the feature selection module as the electrooculogram data after feature processing.

5. A facial expression classification system according to any one of claims 3 or 4, characterized in that: The feature selection module includes a feature fusion unit, an average MIC calculation unit, a feature screening unit and a feature alignment unit; The feature fusion unit receives the feature-processed electromyographic data and electrooculographic data sent by the electromyographic feature processing module and the electrooculographic feature processing module respectively and fuses them to obtain fused features; The average MIC calculation unit calculates the average MIC of the fused features; the feature screening unit screens the fused features according to the calculated average MIC to obtain the screened features; feature The alignment unit aligns the screened features to obtain aligned features; The aligned features are then input into the machine learning expression classification module.

6. A facial expression classification system according to claim 5, characterized in that: The fusion is one or more combinations of addition, concatenation, multi-level fusion and multi-scale fusion.

7. A method for classifying facial expressions, characterized in that: The steps include: S1, collect and obtain facial electromyography signals and electrooculography signals; S2, preprocessing facial electromyographic signals and electrooculographic signals respectively; S3, performing analog-to-digital conversion on the facial electromyography signal and electrooculography signal after signal preprocessing respectively; S4, preprocessing the facial electromyography data and electrooculography data obtained after analog-to-digital conversion; S5, performing feature processing on the facial electromyography data and electrooculography data obtained after data preprocessing respectively; S6, performing feature selection on the facial electromyography features and electrooculography features obtained after feature processing to obtain aligned features; S7. Classify the aligned features obtained after feature selection through a machine learning model to obtain expression types.

8. A method for classifying facial expressions according to claim 7, characterized in that: S2 performs signal preprocessing on the facial electromyographic signal, specifically: performing analog low-pass filtering on the electromyographic signal; performing a first-stage amplification on the electromyographic signal after the analog low-pass filtering; performing analog high-pass filtering on the electromyographic signal after the first-stage amplification; performing a second-stage amplification on the electromyographic signal after the analog high-pass filtering to obtain the facial electromyographic signal after signal preprocessing; performing signal preprocessing on the electrooculogram signal, specifically: performing analog low-pass filtering on the electrooculogram signal; performing a first-stage amplification on the electrooculogram signal after the analog low-pass filtering; performing analog high-pass filtering on the electrooculogram signal after the first-stage amplification; performing a second-stage amplification on the electrooculogram signal after the analog high-pass filtering to obtain the electrooculogram signal after signal preprocessing; S4 performs data preprocessing on the facial electromyography data obtained after analog-to-digital conversion, specifically: digitally filtering the electromyography data after analog-to-digital conversion, normalizing the electromyography data after digital filtering, and obtaining the electromyography data after data preprocessing; performs data preprocessing on the electrooculography data obtained after analog-to-digital conversion, specifically: digitally filtering the electrooculography data after analog-to-digital conversion, normalizing the electrooculography data after digital filtering, and obtaining the electrooculography data after data preprocessing.

9. A method for classifying facial expressions according to claim 7, characterized in that: S5 performs feature processing on the facial electromyography data obtained after data preprocessing, specifically: extracting time domain, frequency domain, statistics and transform domain features of the facial electromyography data after data preprocessing to obtain electromyography extraction features; Calculate the electromyographic correlation coefficient of the electromyographic extraction feature, and determine whether the calculated electromyographic correlation coefficient is less than the electromyographic correlation coefficient threshold. If it is less than the electromyographic correlation coefficient threshold, cross the electromyographic extraction features corresponding to the electromyographic correlation coefficient to obtain the electromyographic data after feature processing; otherwise, if it is greater than or equal to the electromyographic correlation coefficient threshold, use the electromyographic extraction features corresponding to the electromyographic correlation coefficient as the electromyographic data after feature processing; The electrooculogram data obtained after data preprocessing is subjected to feature processing, specifically: time domain, frequency domain, statistical and transform domain features of the electrooculogram data after data preprocessing are extracted to obtain electrooculogram extraction features; Calculate the electrooculogram correlation coefficient of electrooculogram extraction features; Determine whether the correlation coefficient calculated by the electrooculogram correlation coefficient calculation unit is less than the electrooculogram correlation coefficient threshold. If so, cross the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient to obtain the feature-processed electrooculogram data; otherwise, if it is greater than or equal to the electrooculogram correlation coefficient threshold, use the electrooculogram extraction features corresponding to the electrooculogram correlation coefficient as the feature-processed electrooculogram data.

10. A method for classifying facial expressions according to claim 7, characterized in that: S6 performs feature selection on the facial electromyographic features and electrooculographic features obtained after feature processing, specifically: receiving the electromyographic and electrooculographic data after feature processing respectively and fusing them to obtain fused features; Calculate the average MIC of the fused features; select the first N fused features corresponding to the calculated average MIC values ​​as the filtered features; The filtered features are aligned to obtain aligned features; the fusion includes one or more combinations of addition, concatenation, multi-level fusion and multi-scale fusion.

Images