A facial expression recognition device and its recognition method

Abstract

A facial expression recognition device and method are provided. The device includes: a lower facial expression recognition unit, comprising: at least one measuring inertial sensor that contacts the user's lower facial skin to sense the user's lower facial expression; a reference inertial sensor located at a position unaffected by facial expressions and capable of sensing the user's head movement; a processing device for obtaining the user's lower facial expression signal based on data from the at least one measuring inertial sensor and the reference inertial sensor; an upper facial expression recognition unit, comprising: at least three electrodes for contacting the user's upper facial skin to generate electrical signals characterizing electrooculogram (EOG) and electromyogram (EMG) signals; the processing device further comprising a signal processing module for receiving and processing the electrical signals generated by the at least three electrodes to obtain an upper facial expression signal; and a facial expression recognition model for recognizing the user's expression based on the lower facial expression signal and the upper facial expression signal.

Description

Technical Field

[0001] This invention relates to the field of smart devices, and more particularly to a facial expression recognition device and its recognition method. Background Technology

[0002] With the emergence of numerous small, low-power sensing devices, people are able to design more and more smart devices such as smart glasses. In smart glasses, the eye frames, lenses, nose pads, and frames are endowed with intelligent sensing and interaction capabilities. Sensing the user's own facial movements and expressions will help improve the efficiency of remote and virtual communication. Existing smart glasses mainly use two methods for facial expression sensing. One method involves placing a miniature camera in front of the smart glasses. This method can precisely monitor facial expressions, but it is highly privacy-invading, increasing the risk of privacy leaks. Furthermore, miniature cameras consume a lot of power, cannot support real-time monitoring, and are heavy, causing burden for users. The other method uses electromyography (EMG) sensing, but this method requires monitoring the movement of all facial muscles, necessitating a large number of electrodes, whose placement is obvious and affects aesthetics.

[0003] Furthermore, current technologies typically involve placing sensing devices on glasses to monitor facial expressions. However, because the muscles responsible for upper and lower facial expressions are relatively independent, and existing devices often lack the fine-grained perception of lower facial expressions, the results of facial expression perception are inaccurate. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention proposes a facial expression recognition device, comprising:

[0005] The lower facial expression recognition unit includes:

[0006] At least one measuring inertial sensor is in contact with the user's lower facial skin to sense the user's lower facial expressions;

[0007] The reference inertial sensor is located in a position that is unaffected by facial expressions and can sense the user's head movements;

[0008] The processing device obtains the user's lower facial expression signal based on data from the at least one measuring inertial sensor and the reference inertial sensor;

[0009] A facial expression recognition model identifies the user's facial expressions based on the lower facial expression signals, wherein the facial expression recognition model is trained and generated using samples of the lower facial expression signals.

[0010] In one embodiment, the facial expression recognition device further includes an upper facial expression recognition unit, which comprises:

[0011] At least three electrodes are used to contact the user's upper facial skin to generate electrical signals for characterizing electrooculography (EOG) and electromyography (EMG) signals.

[0012] The processing device further includes a signal processing module for receiving and processing electrical signals generated by the at least three electrodes to obtain facial expression signals.

[0013] The facial expression recognition model is also used to recognize the user's expression based on the upper facial expression signal, and the facial expression recognition model is trained using the upper facial expression signal samples.

[0014] In one embodiment, the facial expression recognition device is an eyeglasses device, the eyeglasses device comprising:

[0015] An eye frame, wherein the first electrode and the second electrode are respectively located at the nose pad position of the eye frame, and the third electrode is located at the nose bridge position in the middle of the eye frame.

[0016] The first leg and the second leg are respectively connected to both sides of the eye frame;

[0017] The first extension and the second extension are respectively connected to the first leg and the second leg; wherein, the first measuring inertial sensor of the at least one measuring inertial sensor is located on the first leg or the second leg.

[0018] In one embodiment, the first extension and the second extension have the shape of an ear clip and fit snugly against the back of the user's chin.

[0019] In one embodiment, the third electrode is fixed to the nose bridge by a tower-shaped spring, the third electrode is located on the top of the tower-shaped spring, and the bottom of the tower-shaped spring is fixed to the nose bridge.

[0020] In one embodiment, the facial expression recognition device further includes a data transmission module for receiving data from the signal processing module, the at least one measurement inertial sensor, and the reference inertial sensor, and transmitting the received data to the processing device.

[0021] In one embodiment, the processing device is further configured to remove head motion artifacts based on data from the at least one measuring inertial sensor and the reference inertial sensor in the following manner:

[0022] The data matrix of the reference inertial sensor is mapped to the data matrix of the at least one measurement inertial sensor in real time based on the rotation matrix and the transfer matrix, wherein the rotation matrix and the transfer matrix are obtained by the data matrix of the reference inertial sensor and the data matrix of the at least one measurement inertial sensor when there is no head movement and no facial expression.

[0023] Based on the data matrix of the at least one measuring inertial sensor and the mapped data matrix of the reference inertial sensor, the lower facial expression signal after removing head motion artifacts is obtained.

[0024] The present invention also provides a facial expression recognition method for use in the above-mentioned facial expression recognition device, the method comprising:

[0025] The facial expression recognition device obtains upper facial expression signals and / or lower facial expression signals.

[0026] The upper facial expression signal and / or lower facial expression signal are input into the facial expression recognition model to recognize the user's expression. The facial expression recognition model is obtained by supervised learning training using samples of upper facial expression signals and / or lower facial expression signals and their corresponding known expression units.

[0027] In one embodiment, the step of obtaining facial expression signals through the facial expression recognition device further includes:

[0028] The data matrix of the reference inertial sensor is mapped to the data matrix of the at least one measurement inertial sensor in real time based on the rotation matrix and the transfer matrix, wherein the rotation matrix and the transfer matrix are obtained by the data matrix of the reference inertial sensor and the data matrix of the at least one measurement inertial sensor when there is no head movement and no facial expression.

[0029] Based on the data matrix of the at least one measuring inertial sensor and the mapped data matrix of the reference inertial sensor, the lower facial expression signal after removing head motion artifacts is obtained.

[0030] In one embodiment, the method further includes: inputting the upper facial expression signal and / or the lower facial expression signal into the facial expression recognition model, outputting the values ​​of multiple expression units, and performing facial expression recognition and virtual reconstruction of facial expressions based on the values ​​of the multiple expression units.

[0031] The facial expression recognition device and method of the present invention generate electrical signals representing electrooculography (EOG) and electromyography (EMG) signals through electrodes on the device, and recognize more refined facial expressions through an IMU on the device. The present invention integrates a low-cost, lightweight IMU and electrodes into the facial expression recognition device, and achieves real-time monitoring of facial expressions in a concealed manner. This not only improves the user's wearing experience but also enhances the smart device's ability to perceive the user's own emotions, expanding new human-computer interaction methods. Attached Figure Description

[0032] Figure 1 A schematic diagram of a facial expression recognition device according to an embodiment of the present invention is shown.

[0033] Figure 2 A schematic diagram of wearing a facial expression recognition device according to an embodiment of the present invention is shown.

[0034] Figure 3 A block diagram of a facial expression recognition device according to an embodiment of the present invention is shown.

[0035] Figure 4 A schematic diagram of an artifact removal process according to an embodiment of the present invention is shown.

[0036] Figure 5 A flowchart of a facial expression recognition method for a facial expression recognition device according to an embodiment of the present invention is shown. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments given in this invention are for illustrative purposes only and do not limit the scope of protection of this invention.

[0038] Figure 1 A schematic diagram of a facial expression recognition device according to an embodiment of the present invention is shown. Figure 2 A schematic diagram of wearing a facial expression recognition device according to an embodiment of the present invention is shown, and Figure 3 A block diagram of a facial expression recognition device according to an embodiment of the present invention is shown. The following is in conjunction with... Figures 1-3 The facial expression recognition device of the present invention will be described in detail.

[0039] like Figure 1 and Figure 2 As shown, the facial expression recognition device is implemented in the form of glasses 100, which is convenient to wear on the user's head. However, the present invention is not limited to this. The facial expression recognition device may also have other implementation forms, including but not limited to head-mounted displays, VR glasses, masks, etc.

[0040] The eyeglasses device 100 includes an eye frame 102, a first temple 101A, a second temple 101B, a first extension 105A, and a second extension 105B. The first temple 101A and the second temple 101B are respectively connected to both sides of the eye frame 102. In one embodiment, the first temple 101A and the second temple 101B are fixedly connected to the eye frame 102; in another embodiment, the first temple 101A and the second temple 101B are rotatably connected to the eye frame 102. The first extension 105A and the second extension 105B are respectively connected to the first temple 101A and the second temple 101B. In one embodiment, the first extension 105A and the second extension 105B are respectively connected to the first temple 101A and the second temple 101B via a snap fastener 106, such as... Figure 1 As shown by mark 109 in the figure.

[0041] An eyeglass frame 102 is placed in front of the user's eyes and has three electrodes: a first electrode A, a second electrode B, and a third electrode C. When the user wears the glasses device 100, the three electrodes A, B, and C remain in contact with the skin around the user's eyes. The three electrodes A, B, and C are used to generate electrical signals characterizing electrooculogram (EOG) and electromyography (EMG) signals to monitor the user's upper facial expressions. EOG is a bioelectrical signal caused by the potential difference between the cornea and retina of the eye, and it changes with eye movement. EMG is an electrical signal accompanying muscle contraction. In one embodiment, the three electrodes A, B, and C are bioelectrodes. Since EOG and EMG signals are used to characterize upper facial expressions, they can also be collectively referred to as upper facial expression signals. In this invention, the upper face refers to the area around the eyes, and the lower face refers to the area around the jaw.

[0042] like Figure 1 As shown, the three electrodes A, B, and C are not in a straight line. The first electrode A and the second electrode B are positioned near the nose pad 104 and are designed in the shape of a nose pad. The third electrode C is placed on the bridge of the nose 103 in the middle of the eye frame 102 and is fixed to the bridge of the nose 103 by a tower-shaped spring 107 (as shown in the attached figure). Figure 1As shown by reference numeral 108 in the attached drawing. The third electrode C is located on the top of the tower-shaped spring 107, and the bottom of the tower-shaped spring 107 is fixed to the bridge of the nose 103, thus allowing the third electrode C to elastically contact the user's facial skin via the tower-shaped spring 107. Simultaneously, the tower-shaped spring 107 disperses the force exerted on the third electrode C by the skin to various parts of the eyeglasses device 100, preventing the eyeglasses device 100 from falling off and ensuring that the third electrode C maintains close contact with the skin when the user expresses various facial expressions. In this embodiment, electrodes A, B, and C are placed in the triangular region of the eye frame 102, making the electrodes more concealed and aesthetically pleasing. Furthermore, the triangular region is where the upper facial muscles are relatively concentrated, allowing for the detection of more facial information with fewer electrodes.

[0043] Although the above embodiment uses three electrodes A, B, and C located at the nose pad and bridge of the nose as examples, the invention is not limited thereto. The eyeglass device 100 may include more electrodes, which can be located at any position on the eye frame 102. For example, the electrodes can be located on the frame portion around the lenses of the eye frame 102, as long as the electrodes remain in contact with the skin around the user's eyes when the user wears the device. In one embodiment, the facial expression recognition device includes at least three electrodes that generate electrical signals for characterizing electrooculographic (EOG) and electromyographic (EMG) signals. Preferably, the at least three electrodes are not in a straight line.

[0044] like Figure 3 As shown, according to one embodiment of the present invention, the eyeglasses device 100 further includes a processing device 304 for processing signals, which includes a signal processing module 301, a data transmission module 302, and a processor 303. The signal processing module 301 is used to receive electrical signals detected by three electrodes A, B, and C on the eye frame 102 to obtain upper facial expression signals.

[0045] In one embodiment, the processing device 304 further includes a right foot drive circuit (not shown) for suppressing common-mode voltage generated by the human body. The human body itself can be considered as an antenna, and can generate common-mode voltage due to interference such as household power supply noise. This common-mode voltage can overshadow weak electrical signals, such as electrooculography signals with amplitudes typically only 0.4 to 10 mV.

[0046] The input terminal of the right foot drive circuit is connected to the output terminal of the signal processing module 301 to collect the common-mode voltage of the human body. The output terminal of the right foot drive circuit is connected to the reference electrode on the human body. The common-mode voltage generated by the human body is fed back to the human body through the reference electrode after being fed back in reverse by the right foot drive circuit. The common-mode voltage generated by the human body is suppressed and canceled after being superimposed with the reverse common-mode voltage.

[0047] In one embodiment, the first electrode A, the second electrode B, and the third electrode C are independently configured, and any one electrode can be selected as the reference electrode, with the remaining electrodes used as test electrodes. In another embodiment, the first electrode A, the second electrode B, and the third electrode C can be connected to the right foot drive circuit via switches. When the first electrode A needs to be designated as the reference electrode, simply connect the first electrode A to the drive circuit. According to other embodiments of the present invention, the processing device 304 may also include multiple right foot drive circuits. In one embodiment, the processing device 304 includes three right foot drive circuits, and the first electrode A, the second electrode B, and the third electrode C can be connected to one right foot drive circuit via switches. In another embodiment, the processing device 304 includes two right foot drive circuits, and the second electrode B and the third electrode C can be connected to one right foot drive circuit via switches.

[0048] For example, when the third electrode C is designated as the reference electrode, the upper facial expression signal S of the horizontal channel can be calculated using formula (1). 水平 And when the designated electrode B is used as the reference electrode, the upper facial expression signal S of the vertical channel can be calculated by formula (2). 垂直 :

[0049] S 水平 =S B -S A (1)

[0050] S 垂直 =S C -S A (2)

[0051] The horizontal direction refers to the direction of the line connecting the two eyes, while the vertical direction refers to the direction perpendicular to the horizontal direction within the plane of the eye frame 102. A S is the potential measured on the first electrode A. B It is the potential measured on the second electrode B, and S C This refers to the potential measured on the third electrode C. When the third electrode C is used as a reference electrode, the left-right movement signals of the eyeballs measured by the first electrode A and the second electrode B are obvious, while the up-down movement signals are not obvious. However, when the second electrode B is used as a reference electrode, the up-down movement signals of the eyeballs measured by the first electrode A and the third electrode C are obvious, while the left-right movement signals are not obvious. The above embodiment can obtain upper facial expression signals in both horizontal and vertical channels to obtain more features, making the final facial expression perception more accurate.

[0052] In one embodiment, the signal processing module 301 includes a differential amplifier circuit and a filter circuit, used to sequentially differentially amplify and filter the electrical signals from three electrodes A, B, and C to obtain an upper facial expression signal. The differential amplifier circuit can obtain an amplified upper facial expression signal S from the vertical channel. 垂直 and the amplified horizontal channel of the upper facial expression signal S 水平 When the passband frequency range of the signal is between 0.5 and 10 Hz, the main extracted signal is the electrooculogram (EOG) signal, which can be used to monitor blinking and eye movements. When the passband frequency is greater than 10 Hz, the main extracted signal is the electromyogram (EMG) signal of facial muscle movements, which can be used to monitor facial expressions. According to one embodiment of the present invention, the frequency band of the filtering circuit in the signal processing module 301 is set to 0-25 Hz, and the filtered upper facial expression signal S in the vertical channel... 垂直 and the filtered horizontal channel upper facial expression signal S 水平 It includes both electrooculogram (EOG) and electromyogram (EMG) signals. In one embodiment, the signal processing module 301 can be implemented as a circuit board located on the bridge of the nose 103.

[0053] In one embodiment, the electrodes and signal processing module of the facial expression recognition device can be collectively referred to as the upper facial expression recognition unit.

[0054] Refer again Figure 1 The first extension 105A and the second extension 105B have a shape similar to an ear clip, allowing them to fit snugly against the back of the user's chin. A first inertial measurement unit (IMU) and a second inertial measurement unit (IMU) are respectively disposed on the first extension 105A and the second extension 105B for monitoring facial expressions. The first IMU and the second IMU can be disposed inside or outside the first extension 105A and the second extension 105B, respectively.

[0055] In past research, IMUs have typically been used to calculate the pose and position of objects, but rarely to extract features from subtle signals. For example, in VR headsets, IMUs are used to monitor head movements in real time. IMU data is usually processed using formulas to obtain advanced pose data, but the signal features utilized from the raw data have not been developed. The inventors discovered in experiments that by placing an IMU on the extensions 105A and 105B, the IMU can detect corresponding signals in real time when the wearer expresses various emotions using their face, providing rich information.

[0056] IMU data is also commonly used to estimate head movements. The IMU's output data changes accordingly as the head moves. Therefore, if a user moves their head while making facial expressions, the IMU data will simultaneously contain signal features of both head movements and lower facial expressions. This creates significant interference with lower facial expression feature extraction; this interference signal is called "head movement artifacts."

[0057] To eliminate head motion artifacts, the eyewear device 100 of the present invention is further provided with a reference IMU, which is used to monitor head movements and assist the first IMU and the second IMU in eliminating head motion artifacts. The reference IMU can be located at any position unaffected by facial expressions; it can be located on the head-mounted device or on the user's head. In one embodiment, the reference IMU is located on the first leg 101A.

[0058] like Figure 3 As shown, the glasses device 100 also includes a data transmission module 302 and a processor 303. The data transmission module 302 is connected to the signal processing module 301, the first IMU, the second IMU, and the reference IMU. It receives facial expression signals from the signal processing module 301, as well as data from the first IMU, the second IMU, and the reference IMU, and transmits the received data to the processor 303, such as a microprocessor chip or a computer. The processor 303 can be located inside or outside the device 100. In one embodiment, the data transmission module 302 includes a Bluetooth Low Energy chip (e.g., NRF52840) and its peripheral circuitry, transmitting the received data to the processor via, for example, Bluetooth. In one embodiment, IMU data can be transmitted using the I2C protocol, and facial expression signals can be transmitted using the UART protocol. In another embodiment, the data transmission module 302 may be omitted. The signal processing module 301, the first IMU, the second IMU, and the reference IMU may be directly (wired or wirelessly) connected to the processor. The processor directly receives the facial expression signal from the signal processing module 301, as well as the data from the first IMU, the second IMU, and the reference IMU.

[0059] Processor 303 performs artifact removal on the received IMU data to obtain facial expression signals. According to an embodiment of the present invention, the inventors proposed a spatial mapping artifact removal method, using triaxial accelerometer and triaxial gyroscope data from the IMU output data as a data matrix, wherein the data matrix of the reference IMU is data matrix A, and the data matrix of the first IMU is data matrix B. When there is no head movement and no facial expression, data matrix A and data matrix B are acquired simultaneously, and the rotation matrix R and transition matrix T of the spatial mapping are solved by the following formula (3).

[0060] B = R * A + T (3)

[0061] Using rotation matrix R and transfer matrix T, each point in data matrix A can be mapped to data matrix B in real time. When there is head movement or facial expression, since the head movement is a large-amplitude movement, the head movement signals of the reference IMU and the first IMU are consistent. Therefore, the data matrix D' mapped from the data matrix C of the reference IMU and the data matrix D of the first IMU both contain the same head movement signal. However, subtle facial expressions cause data matrices D' and D to contain different signal characteristics. Therefore, large-amplitude head movements can be removed by subtracting data matrix D' from data matrix D to obtain the lower facial expression signal representing the lower facial expression. Similarly, the lower facial expression signal of the second IMU can be obtained based on the data of the reference IMU and the second IMU. According to the method of the present invention, the signal characteristics of facial expression are not actually the data output of the IMU, but the difference between the output data of the measurement IMU and the reference IMU is used as the lower facial expression signal.

[0062] Although the above embodiments are illustrated using two measurement IMUs (a first IMU and a second IMU) and one reference IMU as examples, the present invention is not limited thereto, and any number of measurement IMUs and reference IMUs can be used. In one embodiment, since facial expressions are generally symmetrical, only one measurement IMU and one reference IMU may be used. In another embodiment, multiple measurement IMUs and multiple reference IMUs may also be used. The location of the measurement IMU is not limited to the extensions 105A and 105B; the measurement IMU may be located at a position where the facial expression recognition device can sense lower facial expressions, such as in contact with the user's cheek. The location of the reference IMU is also not limited to the first leg 101A; the reference IMU may be located at a position unaffected by facial expressions and capable of monitoring the user's head movements, such as the back of the user's head. In one embodiment, the measurement IMUs and reference IMUs of the facial expression recognition device may be collectively referred to as the lower facial expression recognition unit.

[0063] Figure 4 A schematic diagram of an artifact removal process according to an embodiment of the present invention is shown. Figure 4 The curve (1) in the figure is the output data of the 6-axis reference IMU, that is, the curve of one dimension of the data matrix C; Figure 4 Curve (2) in the figure is the curve of the corresponding dimension of the data matrix D' after the data matrix C of the reference IMU is mapped by the rotation matrix R and the transfer matrix T; Figure 4 Curve (3) in the figure is the output data of the first IMU, that is, the curve corresponding to the dimension of the data matrix D; Figure 4 The curve (4) in the figure is the corresponding one-dimensional curve obtained by subtracting the data matrix D' from the data matrix D, that is, the output data of the first IMU after eliminating head artifacts.

[0064] Preferably, the IMUs used in this invention are all of the same model, so their output data have the same scale, that is, the two data matrices have the same scale.

[0065] According to one embodiment of the present invention, a machine learning model is used to perform facial expression recognition on the upper and lower facial data of the wearer obtained in the above embodiments.

[0066] According to an embodiment of the present invention, a method for training a multimodal sensing machine learning model for facial expression recognition is provided. It includes:

[0067] Step 101: Collect upper and lower facial expression signals from multiple volunteers using the facial expression recognition device described above. Simultaneously, extract facial expression videos of the volunteers using a camera, align the video frames with the upper and lower facial expression signals, and obtain a set of known expression units corresponding to each set of upper and lower facial expression signals.

[0068] In one embodiment, to obtain more refined facial expressions, the Facial Behavior Coding System (FACS) is used as a reference standard for facial muscle movement. Expressions are analyzed and classified by recognizing Action Units (AUs). FACS depicts the correspondence between different facial muscle activities and different expressions. Based on the anatomical characteristics of the human face, it divides the face into several independent yet interconnected action units. FACS is now a highly authoritative reference standard for facial muscle movement, enabling the analysis and classification of expressions through the recognition of action units. For example, AU1 represents the raising of the inner eyebrow. The FACS system includes many action units, and any number can be selected as needed in practical applications.

[0069] OpenFace is a mature, open-source framework that uses deep convolutional networks to process facial features. OpenFace can extract and recognize facial expressions in images and output the intensity of 17 motion units (motor units) representing facial expressions, as well as the horizontal angle h and vertical angle v of eye movement. The intensity of a motion unit can be represented from 0 to 5; if the face does not express that motion unit, the intensity is 0; if it expresses the motion unit 100%, the intensity is 5.

[0070] In one embodiment, the intensity of 17 motion units of a volunteer's facial expressions and the horizontal angle h and vertical angle v of eye movement are extracted from the video image using the OpenFace face recognition framework to obtain a set of known expression units corresponding to each set of upper and lower facial expression signals.

[0071] Step 102: Construct an end-to-end multimodal sensing machine learning model with multiple inputs and multiple outputs to extract and recognize features from upper and lower facial expression signals. The model's inputs are upper and lower facial expression signals, and its outputs are the values ​​of multiple expression units.

[0072] In one embodiment, the multiple expression units include the aforementioned 17 motion units, plus horizontal and vertical eye movements, totaling 19 expression units. The values ​​of the multiple expression units refer to the intensity of the 17 motion units and the horizontal angle h and vertical angle v of the eye movements.

[0073] In one embodiment, the machine learning model consists of three convolutional layers and two fully connected layers. The kernel size is 15, the stride is 1, and the activation function is ReLU. The first convolutional layer produces a value of 26*136, the second convolutional layer produces a value of 52*122, and the third convolutional layer produces a value of 104*108. Then, two fully connected layers are applied, finally outputting the values ​​of 19 facial expression units.

[0074] Step 103: Based on the known facial expression units corresponding to the upper and lower facial expression signals, obtain the weights of each layer of the model through supervised learning training.

[0075] Accordingly, the present invention also provides a facial expression recognition method for the above-mentioned facial expression recognition device. Figure 5 A flowchart of the method is shown, which includes:

[0076] Step S201: Collect the user's upper facial expression signals and lower facial expression signals through a facial expression recognition device.

[0077] Step S202: Input the user's upper facial expression signal and lower facial expression signal into the trained multimodal sensing machine learning model and output the values ​​of multiple expression units.

[0078] Step S203: Perform virtual reconstruction of facial expressions and facial expression recognition based on the values ​​of multiple expression units.

[0079] Although the facial expression recognition model in the above embodiments identifies user facial expressions based on upper and lower facial expression signals, the present invention is not limited thereto. Since the upper and lower facial expression signals are relatively independent, the facial expression recognition model in the present invention can also identify user facial expressions based solely on either the upper or lower facial expression signals.

[0080] The facial expression recognition device and method of the present invention generate electrical signals representing electrooculography (EOG) and electromyography (EMG) signals through electrodes on the device, and recognize more refined facial expressions through an IMU on the device. The present invention integrates a low-cost, lightweight IMU and electrodes into the facial expression recognition device, and achieves real-time monitoring of facial expressions in a concealed manner. This not only improves the user's wearing experience but also enhances the smart device's ability to perceive the user's own emotions, expanding new human-computer interaction methods.

[0081] While the present invention has been described through preferred embodiments, it is not limited to the embodiments described herein, and various changes and modifications are made without departing from the scope of the invention.

Claims

1. A facial expression recognition apparatus, comprising: a lower facial expression recognition unit, comprising: at least one measurement inertial sensor in contact with the lower facial skin of a user to sense the lower facial expression of the user; a reference inertial sensor located at a position that is not affected by facial expression and is capable of sensing the head movement of the user; a processing device configured to obtain a lower facial expression signal of the user based on data from the at least one measurement inertial sensor and the reference inertial sensor, comprising: mapping a data matrix of the reference inertial sensor to a data matrix of the at least one measurement inertial sensor in real time based on a rotation matrix and a transfer matrix, wherein the rotation matrix and the transfer matrix are obtained by a data matrix of the reference inertial sensor and a data matrix of the at least one measurement inertial sensor when there is no head movement and no expression; obtaining a lower facial expression signal with head movement artifacts removed based on the data matrix of the at least one measurement inertial sensor and the mapped data matrix of the reference inertial sensor; a facial expression recognition model configured to recognize an expression of the user based on the lower facial expression signal, wherein the facial expression recognition model is trained using samples of the lower facial expression signal.

2. The facial expression recognition apparatus of claim 1, wherein, The facial expression recognition apparatus further comprises an upper facial expression recognition unit, comprising: at least three electrodes in contact with the upper facial skin of the user to generate electrical signals for characterizing electro-oculogram signals and electromyogram signals; the processing device further comprises a signal processing module configured to receive and process the electrical signals generated by the at least three electrodes to obtain an upper facial expression signal; wherein the facial expression recognition model is further configured to recognize the expression of the user based on the upper facial expression signal, wherein the facial expression recognition model is further trained using samples of the upper facial expression signal.

3. The facial expression recognition apparatus of claim 2, wherein, The facial expression recognition apparatus is a glasses apparatus, comprising: an eye frame, wherein a first electrode and a second electrode are located at a nose pad position of the eye frame, and a third electrode is located at a nose bridge position in the middle of the eye frame; a first leg and a second leg connected to two sides of the eye frame, respectively; a first extension and a second extension connected to the first leg and the second leg, respectively; wherein a first measurement inertial sensor of the at least one measurement inertial sensor is located on the first leg or the second leg.

4. The facial expression recognition apparatus of claim 3, wherein, The first extension and the second extension have the shape of an ear clip and are close to the back of the lower jaw of the user.

5. The facial expression recognition apparatus of claim 3, wherein, The third electrode is fixed to the nose bridge by a tower-shaped spring, the third electrode is located at the top of the tower-shaped spring, and the bottom of the tower-shaped spring is fixed to the nose bridge.

6. The facial expression recognition apparatus of claim 2, wherein, The facial expression recognition apparatus further comprises a data transmission module configured to receive data from the signal processing module, the at least one measurement inertial sensor, and the reference inertial sensor, and transmit the received data to the processing device. 7.A facial expression recognition method for the facial expression recognition apparatus of any one of claims 2-6, the method comprising: obtaining an upper facial expression signal and / or a lower facial expression signal by the facial expression recognition apparatus. inputting the upper facial expression signal and / or the lower facial expression signal into the facial expression recognition model to recognize the user expression, wherein the facial expression recognition model is obtained by training through supervised learning using the upper facial expression signal and / or the lower facial expression signal and a sample of a known expression unit corresponding thereto.

8. The facial expression recognition method of claim 7, wherein, The method further comprises: inputting the upper facial expression signal and / or the lower facial expression signal into the facial expression recognition model to output values of a plurality of expression units, performing facial expression recognition and facial expression virtual reconstruction based on the values of the plurality of expression units.

Images