Emotion recognition method and system based on multi-modal biological characteristics

Abstract

The invention discloses an emotion recognition method and system based on multi-modal biological characteristics, and relates to the technical field of emotion recognition, and the system comprises a multi-modal module, a calculation module, a dynamic module and a summary module. A coupling unit on the summarizing module also calculates a first coupling degree between the eye movement features and the micro-expression features and a second coupling degree between the eye movement features and the voice features through the eye movement feature fluctuation degree, the micro-expression feature fluctuation degree, the voice feature fluctuation degree and a coupling formula respectively, and when the value of the first coupling degree is closer to 1, the second coupling degree is closer to 1; the micro-expression features and the eye movement features are more synchronous, the micro-expression features and the eye movement features of the tested person are closer to natural physiological reactions, and the micro-expression features are not deliberately controlled; when the numerical value of the second coupling degree is closer to 1, the voice features and the eye movement features are more synchronous, further, the voice features and the eye movement features of the testee are closer to natural physiological reactions, and the voice features are not deliberately controlled.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and in particular to an emotion recognition method and system based on multimodal biometrics. Background Technology

[0002] Biometric emotion recognition primarily uses non-invasive sensors to collect various real-time data, such as facial expressions, voice tone, brain waves, heart rate, skin conductance, and even body temperature. Artificial intelligence algorithms are then used to model and correlate these biosignals with specific emotional patterns. Furthermore, since most biometric signals are regulated by the autonomic nervous system and are not easily controlled by an individual, they can provide more objective, continuous, and realistic emotional feedback. Its applications are widespread in several cutting-edge fields: in mental health, it can assist in the diagnosis of depression or post-traumatic stress disorder; in human-computer interaction, it enables smart devices to understand and respond to user needs more naturally; and in security monitoring, it helps identify abnormal emotional states to prevent risks.

[0003] The existing biometric emotion recognition technologies often have the following problems: (1) Existing multimodal emotion recognition methods usually use fixed weights or simple training weights, which cannot distinguish between "controllable expressive features" and "uncontrollable physiological features", resulting in serious distortion of recognition results when subjects deliberately make facial expressions or voice disguises, and cannot automatically generate weights from the changes in the features themselves; (2) Existing technologies only focus on a single modality or simple fusion, which cannot identify "emotional disguise" scenarios, and cannot identify and analyze the coupling degree between eye movements, facial expressions and voice behaviors; (3) Existing emotion recognition is mostly result recognition, which cannot determine whether the emotion is generated naturally or deliberately expressed, and cannot identify the generation path of real emotions by judging the response order of different modalities on the time axis. Summary of the Invention

[0004] The purpose of this invention is to provide an emotion recognition method and system based on multimodal biometrics, aiming to solve the technical problems existing in the prior art, such as how to automatically generate weights from the changes in the feature itself, how to identify and analyze the coupling degree between eye movement, facial expression and voice behavior, and how to identify the generation path of real emotions by judging the response order of different modalities on the time axis.

[0005] To address the aforementioned technical problems, the present invention adopts the following technical solution: an emotion recognition method based on multimodal biometrics, comprising the following steps:

[0006] Step S1: First, calibrate and adjust the microphone, eye-tracking device and facial capture device in the test unit. Then, the subject begins to describe his own emotionally moving experiences in front of the microphone, eye-tracking device and facial capture device. The emotionally moving experiences include at least one experience of sadness, one experience of shock, one experience of anger and one experience of happiness.

[0007] Step S2: During the process of the subject recounting an emotionally moving experience, the eye-tracking device will collect the subject's eye movement feature data, the facial capture device will collect micro-expression feature data, and the microphone will collect voice feature data.

[0008] Step S3: Eye movement feature data including fixation duration Number of fixations and the number of eye twitches Micro-expression feature data includes the number of brow muscle movements. Eyelid contraction frequency and the number of times the corners of the mouth change Speech feature data includes the number of fundamental frequency changes in speech. Number of changes in speech intensity and the number of speech rate changes ;

[0009] Step S4: The eye-tracking unit records the fixation duration. Number of fixations and the number of eye twitches Substituting these values ​​into the characteristic volatility formula, we can calculate the eye movement characteristic volatility. The micro-expression unit then moves the brow muscles a number of times. Eyelid contraction frequency and the number of times the corners of the mouth change Substituting these values ​​into the feature volatility formula, we can calculate the micro-expression feature volatility. The speech unit then records the number of fundamental frequency changes. Number of changes in speech intensity and the number of speech rate changes Substituting these values ​​into the feature fluctuation formula, we can calculate the speech feature fluctuation. ;

[0010] Step S5: Then, the eye movement feature variability is calculated. Micro-expression feature fluctuation Speech feature fluctuation The data is sent to the calculation module, which then calculates the eye-tracking reliability parameters using the reliability calculation formula. Micro-expression credibility parameters and speech credibility parameters ;

[0011] Step S6: Subsequent eye-tracking reliability parameters Micro-expression credibility parameters and speech credibility parameters The output will also be sent to the dynamic module, which will calculate the eye-tracking weights using the modality proportion formula. ;

[0012] Step S7: While performing step S6, the coupling unit on the summary module also uses eye-tracking feature variability... Micro-expression feature fluctuation Speech feature fluctuation The coupling formulas are used to calculate the first degree of coupling between eye-tracking features and micro-expression features, respectively. The second coupling degree between eye movement features and speech features ;

[0013] Step S8: The detection unit determines the weights based on eye movement. First degree of coupling Second coupling degree The emotional consistency formula is used to calculate the true emotional consistency. Meanwhile, the detection unit also calculates the leading factor using the emotion leading formula. The subsequent judgment unit determined the consistency of the actual emotions. and leading factor Calculate the credibility index of true emotion .

[0014] Preferably, the acquisition time periods of the eye-tracking device, the face acquisition device, and the microphone are all the same time period; the acquisition time period is evenly divided into... Statistical time period The value ranges from 60 to 100; fixation duration Indicates the first Total gaze duration within a statistical time period The value range is 1 to Number of fixations Indicates the first Total number of fixations within a statistical time period; number of saccades Indicates the first Total number of eye twitches within a statistical time period; number of eyebrow muscle movements Indicates the first The total number of times the brow muscles shifted within a statistical time period; the number of times the eyelids contracted. Indicates the first Total number of eyelid contractions within a statistical time period; number of changes in the corners of the mouth. Indicates the first The total number of changes in the corners of the mouth within a statistical time period; the number of changes in the fundamental frequency of the voice. Indicates the first The total number of times the fundamental frequency of speech changed within a statistical time period; the number of times the speech intensity changed. Indicates the first Total number of times speech intensity changed within a statistical time period; number of times speech rate changed. Indicates the first The total number of times the speech rate changed within a statistical time period.

[0015] Preferably, the characteristic volatility formula is as follows:

[0016] ;

[0017] In the formula: express gaze duration The average value; express Number of fixations The average value; express twitching count The average value; express Number of eyebrow muscle displacements The average value; express Number of eyelid contractions The average value; express Number of changes in the corners of the mouth The average value; express Number of changes in the fundamental frequency of the voice The average value; express Number of voice intensity changes The average value; express Number of speech rate changes The average value.

[0018] Preferably, the credibility calculation formula is as follows:

[0019] ;

[0020] The modal proportion formula is as follows:

[0021] ;

[0022] In the formula: Indicates the first proportion weight. The value range is 1.8 to 2; Indicates the second proportion weight. The value range is 1.2 to 1.7; Indicates the third proportion weight. The value range is 1 to 1.2; when the eye-tracking reliability parameter The larger the value, the greater the variability of eye movement features. The smaller the value, the closer the subject's eye movement characteristics are to natural physiological responses, rather than exaggerated or artificial ones; when the micro-expression credibility parameter... The larger the value, the greater the volatility of micro-expression features. The smaller the value, the closer the micro-expression features of the test subjects are to their natural physiological responses; when the voice credibility parameter... The larger the value, the greater the speech feature variability. The smaller the value, the closer the subject's speech characteristics are to their natural physiological responses.

[0023] Preferably, the coupling formula is used to calculate the first coupling degree. Second Coupling Degree In all cases, eye movement features are used as the basis; the coupling formula is shown below:

[0024] ;

[0025] When the first coupling degree The closer the value is to 1, the more synchronized the micro-expression features and eye movement features are, further indicating that the micro-expression features and eye movement features of the subjects are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, further indicating that the subjects' micro-expression features and eye movement features are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, and the more synchronized they ... and the more synchronized they are, and the more synchronized they are, The closer the value is to 1, the more synchronized the speech features and eye movement features are, further indicating that the speech features and eye movement features of the subjects are closer to natural physiological responses and that the speech features were not deliberately controlled.

[0026] Preferably, the formula for emotional consistency is as follows:

[0027] ;

[0028] Let the time interval from the start of eye-tracking device acquisition to the first significant change in eye movement feature data be the eye response time. The micro-expression response time is the time interval from when the facial capture device starts collecting data until the first significant change occurs in the micro-expression feature data. The microphone response time is the time interval from when the microphone starts collecting data until the first significant change occurs in the speech feature data. The emotion-leading formula is shown below:

[0029] ;

[0030] In the formula: Represents the natural constant; emotions in the central nervous system first affect eye movements, fixation, and pupils, and then affect facial muscles and the speech system; when the leading factor When the value is less than 1, it indicates that the microphone, eye-tracking device, and facial recognition device are damaged; when the leading factor... When the value is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; when the leading factor is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; When the value is greater than 1.2 and less than 1.5, it indicates that eye-movement features slightly precede micro-expression features and speech features, and the corresponding emotion may be mixed with controlled behavior; when the leading factor When the value is greater than or equal to 1.5, it indicates that eye movement features significantly lead micro-expression features and speech features, and the corresponding emotions are naturally triggered by internal psychological states to manifest behavior.

[0031] Preferably, the credibility index of genuine emotions The calculation formula is as follows:

[0032] ;

[0033] In the formula: when the credibility index of true emotion is... A higher value indicates that the emotions expressed by the subjects are more authentic and credible. Using eye-tracking feature data as a baseline, the credibility index of authentic emotions is further enhanced through comparative coupling between eye-tracking feature data and micro-expression feature data, and through comparative coupling between eye-tracking feature data and speech feature data. The final judgment is more accurate.

[0034] The present invention also provides an emotion recognition system based on multimodal biometrics, including a multimodal module, a calculation module, a dynamic module, and a summary module; the multimodal module and the calculation module are unidirectionally connected; the calculation module is unidirectionally connected to the dynamic module and the summary module respectively; the dynamic module and the summary module are unidirectionally connected.

[0035] Preferably, the multimodal module includes a testing unit, an eye-tracking unit, a micro-expression unit, and a speech unit; the testing unit is unidirectionally connected to the eye-tracking unit, the micro-expression unit, and the speech unit respectively; the testing unit is also equipped with a microphone, an eye-tracking device, and a facial acquisition device; the eye-tracking unit is unidirectionally connected to the computing module; the micro-expression unit is unidirectionally connected to the computing module; and the speech unit is unidirectionally connected to the computing module.

[0036] Preferably, the summary module includes a coupling unit, a detection unit, and a judgment unit; the coupling unit and the detection unit are unidirectionally connected; the detection unit and the judgment unit are unidirectionally connected; and the dynamic module and the detection unit are unidirectionally connected.

[0037] The beneficial effects of this invention compared with the prior art are: (1) Eye movement feature variability Micro-expression feature fluctuation Speech feature fluctuation The data is sent to the calculation module, which then calculates the eye-tracking reliability parameters using the reliability calculation formula. Micro-expression credibility parameters and speech credibility parameters Subsequently, eye-tracking reliability parameters Micro-expression credibility parameters and speech credibility parameters The output will also be sent to the dynamic module, which will calculate the eye-tracking weights using the modality proportion formula. , , , The higher the value, the more natural the emotional expression of the subjects. The value will also change accordingly; (2) The coupling unit on the summary module also uses eye movement feature fluctuation. Micro-expression feature fluctuation Speech feature fluctuation The coupling formulas are used to calculate the first degree of coupling between eye-tracking features and micro-expression features, respectively. The second coupling degree between eye movement features and speech features When the first coupling degree The closer the value is to 1, the more synchronized the micro-expression features and eye movement features are, further indicating that the micro-expression features and eye movement features of the subjects are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, further indicating that the subjects' micro-expression features and eye movement features are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, and the more synchronized they ... and the more synchronized they are, and the more synchronized they are, The closer the value is to 1, the more synchronized the speech features and eye movement features are, further indicating that the speech features and eye movement features of the subjects are closer to natural physiological responses and that there is no deliberate control of speech features. (3) The detection unit calculates the leading factor through the emotion leading formula. When the leading factor When the value is less than 1, it indicates that the microphone, eye-tracking device, and facial recognition device are damaged; when the leading factor... When the value is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; when the leading factor is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; When the value is greater than 1.2 and less than 1.5, it indicates that eye-movement features slightly precede micro-expression features and speech features, and the corresponding emotion may be mixed with controlled behavior; when the leading factor When the value is greater than or equal to 1.5, it indicates that eye movement features significantly lead micro-expression features and speech features, and the corresponding emotions are naturally triggered by internal psychological states to manifest behavior. Attached Figure Description

[0038] Figure 1 This is a flowchart of the emotion recognition method based on multimodal biometrics according to the present invention.

[0039] Figure 2 This is a schematic diagram of the overall architecture of the emotion recognition system based on multimodal biometrics of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the specific structural principle of the emotion recognition system based on multimodal biometrics according to the present invention.

[0041] In the diagram: 1-Processing module; 2-First detection module; 3-Second detection module; 4-Correction module; 5-Deduction module; 11-Pulverizing unit; 12-Extraction unit; 13-Extraction unit; 41-Denoising unit; 42-Statistical unit; 51-Calculation unit; 52-Output unit. Detailed Implementation

[0042] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0044] Figures 1 to 3 This is a preferred embodiment of the present invention.

[0045] Figure 1 A flowchart of the emotion recognition method based on multimodal biometrics of the present invention is provided, including the following steps:

[0046] Step S1: First, calibrate and adjust the microphone, eye-tracking device and facial capture device in the test unit 11. Then, the subject begins to describe his own emotionally moving experiences in front of the microphone, eye-tracking device and facial capture device. The emotionally moving experiences include at least one experience of sadness, one experience of shock, one experience of anger and one experience of happiness.

[0047] Step S2: During the process of the subject recounting an emotionally moving experience, the eye-tracking device will collect the subject's eye movement feature data, the facial capture device will collect micro-expression feature data, and the microphone will collect voice feature data.

[0048] Step S3: Eye movement feature data including fixation duration Number of fixations and the number of eye twitches Micro-expression feature data includes the number of brow muscle movements. Eyelid contraction frequency and the number of times the corners of the mouth change Speech feature data includes the number of fundamental frequency changes in speech. Number of changes in speech intensity and the number of speech rate changes ;

[0049] Step S4: Eye-tracking unit 12 records fixation duration Number of fixations and the number of eye twitches Substituting these values ​​into the characteristic volatility formula, we can calculate the eye movement characteristic volatility. Micro-expression unit 13 then moves the brow muscles a number of times. Eyelid contraction frequency and the number of times the corners of the mouth change Substituting these values ​​into the feature volatility formula, we can calculate the micro-expression feature volatility. Subsequently, the speech unit 14 will record the number of fundamental frequency changes. Number of changes in speech intensity and the number of speech rate changes Substituting these values ​​into the feature fluctuation formula, we can calculate the speech feature fluctuation. ;

[0050] Step S5: Then, the eye movement feature variability is calculated. Micro-expression feature fluctuation Speech feature fluctuation The data is sent to calculation module 2, which then calculates the eye-tracking reliability parameters using the reliability calculation formula. Micro-expression credibility parameters and speech credibility parameters ;

[0051] Step S6: Subsequent eye-tracking reliability parameters Micro-expression credibility parameters and speech credibility parameters The output will also be sent to dynamic module 3, which will calculate the eye-tracking weights using the modality proportion formula. ;

[0052] Step S7: While performing step S6, the coupling unit 41 on the summarizing module 4 also uses eye-tracking feature fluctuations. Micro-expression feature fluctuation Speech feature fluctuation The coupling formulas are used to calculate the first degree of coupling between eye-tracking features and micro-expression features, respectively. The second coupling degree between eye movement features and speech features ;

[0053] Step S8: Detection unit 42 determines the weights based on eye movement. First degree of coupling Second coupling degree The emotional consistency formula is used to calculate the true emotional consistency. Meanwhile, the detection unit 42 also calculates the leading factor using the emotion leading formula. Subsequently, Unit 43 determined the consistency of the actual emotions. and leading factor Calculate the credibility index of true emotion .

[0054] The working principle of this invention is as follows: First, the microphone, eye-tracking device, and facial recognition device in the testing unit 11 are calibrated and adjusted. Then, the subject begins to describe their own emotionally moving experiences in front of the microphone, eye-tracking device, and facial recognition device. These emotionally moving experiences include at least one experience of sadness, one experience of shock, one experience of anger, and one experience of happiness. During the subject's description of these emotionally moving experiences, the eye-tracking device collects eye movement feature data, the facial recognition device collects micro-expression feature data, and the microphone collects voice feature data. The eye movement feature data includes the duration of fixation. Number of fixations and the number of eye twitches Micro-expression feature data includes the number of brow muscle movements. Eyelid contraction frequency and the number of times the corners of the mouth change Speech feature data includes the number of fundamental frequency changes in speech. Number of changes in speech intensity and the number of speech rate changes Eye-tracking unit 12 will determine the duration of fixation. Number of fixations and the number of eye twitches Substituting these values ​​into the characteristic volatility formula, we can calculate the eye movement characteristic volatility. Micro-expression unit 13 then moves the brow muscles a number of times. Eyelid contraction frequency and the number of times the corners of the mouth change Substituting these values ​​into the feature volatility formula, we can calculate the micro-expression feature volatility. Subsequently, the speech unit 14 will record the number of fundamental frequency changes. Number of changes in speech intensity and the number of speech rate changes Substituting these values ​​into the feature fluctuation formula, we can calculate the speech feature fluctuation. Subsequently, the eye movement feature variability was measured. Micro-expression feature fluctuation Speech feature fluctuation The data is sent to calculation module 2, which then calculates the eye-tracking reliability parameters using the reliability calculation formula. Micro-expression credibility parameters and speech credibility parameters Subsequently, eye-tracking reliability parameters Micro-expression credibility parameters and speech credibility parameters The output will also be sent to dynamic module 3, which will calculate the eye-tracking weights using the modality proportion formula. Simultaneously with step S6, the coupling unit 41 on the summarizing module 4 also uses eye-tracking feature fluctuations. Micro-expression feature fluctuation Speech feature fluctuation The coupling formulas are used to calculate the first degree of coupling between eye-tracking features and micro-expression features, respectively. The second coupling degree between eye movement features and speech features ;Detection unit 42 based on eye movement weights First degree of coupling Second coupling degree The emotional consistency formula is used to calculate the true emotional consistency. Meanwhile, the detection unit 42 also calculates the leading factor using the emotion leading formula. Subsequently, Unit 43 determined the consistency of the actual emotions. and leading factor Calculate the credibility index of true emotion .

[0055] Furthermore, the data acquisition time periods of the eye-tracking device, the face capture device, and the microphone are all the same time period; the acquisition time period is evenly divided into... Statistical time period The value ranges from 60 to 100; fixation duration Indicates the first Total gaze duration within a statistical time period The value range is 1 to Number of fixations Indicates the first Total number of fixations within a statistical time period; number of saccades Indicates the first Total number of eye twitches within a statistical time period; number of eyebrow muscle movements Indicates the first The total number of times the brow muscles shifted within a statistical time period; the number of times the eyelids contracted. Indicates the first Total number of eyelid contractions within a statistical time period; number of changes in the corners of the mouth. Indicates the first The total number of changes in the corners of the mouth within a statistical time period; the number of changes in the fundamental frequency of the voice. Indicates the first The total number of times the fundamental frequency of speech changed within a statistical time period; the number of times the speech intensity changed. Indicates the first Total number of times speech intensity changed within a statistical time period; number of times speech rate changed. Indicates the first The total number of times the speech rate changed within a statistical time period.

[0056] Furthermore, the characteristic volatility formula is as follows:

[0057] ;

[0058] In the formula: express gaze duration The average value; express Number of fixations The average value; express twitching count The average value; express Number of eyebrow muscle displacements The average value; express Number of eyelid contractions The average value; express Number of changes in the corners of the mouth The average value; express Number of changes in the fundamental frequency of the voice The average value; express Number of voice intensity changes The average value; express Number of speech rate changes The average value.

[0059] Furthermore, the credibility calculation formula is as follows:

[0060] ;

[0061] The modal proportion formula is as follows:

[0062] ;

[0063] In the formula: Indicates the first proportion weight. The value range is 1.8 to 2; Indicates the second proportion weight. The value range is 1.2 to 1.7; Indicates the third proportion weight. The value range is 1 to 1.2; when the eye-tracking reliability parameter The larger the value, the greater the variability of eye movement features. The smaller the value, the closer the subject's eye movement characteristics are to natural physiological responses, rather than exaggerated or artificial ones; when the micro-expression credibility parameter... The larger the value, the greater the volatility of micro-expression features. The smaller the value, the closer the micro-expression features of the test subjects are to their natural physiological responses; when the voice credibility parameter... The larger the value, the greater the speech feature variability. The smaller the value, the closer the subject's speech characteristics are to their natural physiological responses.

[0064] Furthermore, the coupling formula is used to calculate the first degree of coupling. Second Coupling Degree In all cases, eye movement features are used as the basis; the coupling formula is shown below:

[0065] ;

[0066] When the first coupling degree The closer the value is to 1, the more synchronized the micro-expression features and eye movement features are, further indicating that the micro-expression features and eye movement features of the subjects are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, further indicating that the subjects' micro-expression features and eye movement features are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, and the more synchronized they ... and the more synchronized they are, and the more synchronized they are, The closer the value is to 1, the more synchronized the speech features and eye movement features are, further indicating that the speech features and eye movement features of the subjects are closer to natural physiological responses and that the speech features were not deliberately controlled.

[0067] Furthermore, the formula for emotional consistency is as follows:

[0068] ;

[0069] Let the time interval from the start of eye-tracking device acquisition to the first significant change in eye movement feature data be the eye response time. The micro-expression response time is the time interval from when the facial capture device starts collecting data until the first significant change occurs in the micro-expression feature data. The microphone response time is the time interval from when the microphone starts collecting data until the first significant change occurs in the speech feature data. The emotion-leading formula is shown below:

[0070] ;

[0071] In the formula: Represents the natural constant; emotions in the central nervous system first affect eye movements, fixation, and pupils, and then affect facial muscles and the speech system; when the leading factor When the value is less than 1, it indicates that the microphone, eye-tracking device, and facial recognition device are damaged; when the leading factor... When the value is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; when the leading factor is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; When the value is greater than 1.2 and less than 1.5, it indicates that eye-movement features slightly precede micro-expression features and speech features, and the corresponding emotion may be mixed with controlled behavior; when the leading factor When the value is greater than or equal to 1.5, it indicates that eye movement features significantly lead micro-expression features and speech features, and the corresponding emotions are naturally triggered by internal psychological states to manifest behavior.

[0072] Furthermore, the credibility index of genuine emotions The calculation formula is as follows:

[0073] ;

[0074] In the formula: when the credibility index of true emotion is... A higher value indicates that the emotions expressed by the subjects are more authentic and credible. Using eye-tracking feature data as a baseline, the credibility index of authentic emotions is further enhanced through comparative coupling between eye-tracking feature data and micro-expression feature data, and through comparative coupling between eye-tracking feature data and speech feature data. The final judgment is more accurate.

[0075] For emotion recognition systems based on multimodal biometrics, such as Figure 2 and Figure 3As shown, the module includes a multimodal module 1, a calculation module 2, a dynamic module 3, and a summary module 4. The multimodal module 1 is unidirectionally connected to the calculation module 2. The calculation module 2 is unidirectionally connected to both the dynamic module 3 and the summary module 4. The dynamic module 3 is unidirectionally connected to the summary module 4. The multimodal module 1 includes a test unit 11, an eye-tracking unit 12, a micro-expression unit 13, and a speech unit 14. The test unit 11 is unidirectionally connected to the eye-tracking unit 12, the micro-expression unit 13, and the speech unit 14. The test unit 11 also includes a microphone, an eye-tracking device, and a facial recognition device. The eye-tracking unit 12 is unidirectionally connected to the calculation module 2. The micro-expression unit 13 is unidirectionally connected to the calculation module 2. The speech unit 14 is unidirectionally connected to the calculation module 2. The summary module 4 includes a coupling unit 41, a detection unit 42, and a judgment unit 43. The coupling unit 41 is unidirectionally connected to the detection unit 42. The detection unit 42 is unidirectionally connected to the judgment unit 43. The dynamic module 3 is unidirectionally connected to the detection unit 42.

[0076] This invention is not limited to the specific embodiments described above. Any modifications made by those skilled in the art based on the above concept without creative effort are within the protection scope of this invention.

Claims

1. An emotion recognition method based on multimodal biometrics, characterized in that, Includes the following steps: Step S1: First, the microphone, eye-tracking device and face acquisition device in the test unit (11) are calibrated and adjusted. Then, the subject begins to tell the subject's own emotionally touching experience in front of the microphone, eye-tracking device and face acquisition device. The emotionally touching experience includes at least one sad experience, one shock experience, one angry experience and one happy experience. Step S2: During the process of the subject recounting an emotionally moving experience, the eye-tracking device will collect the subject's eye movement feature data, the facial capture device will collect micro-expression feature data, and the microphone will collect voice feature data. Step S3: Eye movement feature data including fixation duration Number of fixations and the number of eye twitches Micro-expression feature data includes the number of brow muscle movements. Eyelid contraction frequency and the number of times the corners of the mouth change Speech feature data includes the number of fundamental frequency changes in speech. Number of changes in speech intensity and the number of speech rate changes ; Step S4: The eye-tracking unit (12) records the fixation duration. Number of fixations and the number of eye twitches Substituting these values ​​into the characteristic volatility formula, we can calculate the eye movement characteristic volatility. The micro-expression unit (13) then moves the brow muscles a number of times. Eyelid contraction frequency and the number of times the corners of the mouth change Substituting these values ​​into the feature volatility formula, we can calculate the micro-expression feature volatility. Subsequently, the speech unit (14) will record the number of fundamental frequency changes. Number of changes in speech intensity and the number of speech rate changes Substituting these values ​​into the feature fluctuation formula, we can calculate the speech feature fluctuation. ; Step S5: Then, the eye movement feature variability is calculated. Micro-expression feature fluctuation Speech feature fluctuation The data is sent to the calculation module (2), which then calculates the eye-tracking reliability parameters using the reliability calculation formula. Micro-expression credibility parameters and speech credibility parameters ; Step S6: Subsequent eye-tracking reliability parameters Micro-expression credibility parameters and speech credibility parameters The output will be simultaneously sent to the dynamic module (3), which calculates the eye-tracking weights using the modality proportion formula. ; Step S7: While performing step S6, the coupling unit (41) on the summarizing module (4) also uses eye-tracking feature fluctuations. Micro-expression feature fluctuation Speech feature fluctuation The coupling formulas are used to calculate the first degree of coupling between eye-tracking features and micro-expression features, respectively. The second coupling degree between eye movement features and speech features ; Step S8: The detection unit (42) determines the weights based on eye movement. First degree of coupling Second coupling degree The emotional consistency formula is used to calculate the true emotional consistency. Meanwhile, the detection unit (42) also calculates the leading factor through the emotion leading formula. The subsequent judgment unit (43) determines the consistency of the actual emotion. and leading factor Calculate the credibility index of true emotion .

2. The emotion recognition method based on multimodal biometrics as described in claim 1, characterized in that: The data acquisition time periods for the eye-tracking device, the face capture device, and the microphone are all the same; the acquisition time period is evenly divided into... Statistical time period The value ranges from 60 to 100; fixation duration Indicates the first Total gaze duration within a statistical time period The value range is 1 to Number of fixations Indicates the first Total number of fixations within a statistical time period; number of saccades Indicates the first Total number of eye twitches within a statistical time period; number of eyebrow muscle movements Indicates the first The total number of times the brow muscles shifted within a statistical time period; the number of times the eyelids contracted. Indicates the first Total number of eyelid contractions within a statistical time period; number of changes in the corners of the mouth. Indicates the first The total number of changes in the corners of the mouth within a statistical time period; the number of changes in the fundamental frequency of the voice. Indicates the first The total number of times the fundamental frequency of speech changed within a statistical time period; the number of times the speech intensity changed. Indicates the first Total number of times speech intensity changed within a statistical time period; number of times speech rate changed. Indicates the first The total number of times the speech rate changed within a statistical time period.

3. The emotion recognition method based on multimodal biometrics as described in claim 2, characterized in that: Characteristic fluctuations The formula for degree is as follows: ； In the formula: express gaze duration The average value; express Number of fixations The average value; express twitching count The average value; express Number of eyebrow muscle movements The average value; express Number of eyelid contractions The average value; express Number of changes in the corners of the mouth The average value; express Number of changes in the fundamental frequency of the voice The average value; express Number of voice intensity changes The average value; express Number of speech rate changes The average value.

4. The emotion recognition method based on multimodal biometrics as described in claim 3, characterized in that: The formula for calculating credibility is as follows: ； The modality proportion formula is as follows: ； In the formula: Indicates the first proportion weight. The value ranges from 1.8 to 2; Indicates the second proportion weight. The value range is 1.2 to 1.7; Indicates the third proportion weight. The value range is 1 to 1.2; when the eye-tracking reliability parameter The larger the value, the greater the variability of eye movement features. The smaller the value, the closer the subject's eye movement characteristics are to natural physiological responses, rather than exaggerated or artificial ones; when the micro-expression credibility parameter... The larger the value, the greater the volatility of micro-expression features. The smaller the value, the closer the micro-expression features of the test subjects are to their natural physiological responses; when the voice credibility parameter... The larger the value, the greater the speech feature variability. The smaller the value, the closer the subject's speech characteristics are to their natural physiological responses.

5. The emotion recognition method based on multimodal biometrics as described in claim 4, characterized in that: The coupling formula is used to calculate the first degree of coupling. Second Coupling Degree In all cases, eye movement characteristics are used as the basis; The coupling formula is shown below: ； When the first coupling degree The closer the value is to 1, the more synchronized the micro-expression features and eye movement features are, further indicating that the micro-expression features and eye movement features of the subjects are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, further indicating that the subjects' micro-expression features and eye movement features are closer to natural physiological responses, and that there is no deliberate control of micro-expression features; when the second coupling degree is closer to 1, the more synchronized the micro-expression features and eye movement features are, the more synchronized they are, and the more synchronized they ... and the more synchronized they are, and the more synchronized they are, The closer the value is to 1, the more synchronized the speech features and eye movement features are, further indicating that the speech features and eye movement features of the subjects are closer to natural physiological responses and that the speech features were not deliberately controlled.

6. The emotion recognition method based on multimodal biometrics as described in claim 5, characterized in that: The formula for emotional consistency is as follows: ； Let the time interval from the start of eye-tracking device acquisition to the first significant change in eye movement feature data be the eye response time. The micro-expression response time is the time interval from when the facial capture device starts collecting data until the first significant change occurs in the micro-expression feature data. The time interval from when the microphone starts collecting data to when the first significant change occurs in the speech feature data is called the microphone response time. The emotion-leading formula is shown below: ； In the formula: Represents the natural constant; emotions in the central nervous system first affect eye movements, fixation, and pupils, and then affect facial muscles and the speech system; when the leading factor When the value is less than 1, it indicates that the microphone, eye-tracking device, and facial recognition device are damaged; when the leading factor... When the value is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; when the leading factor is greater than or equal to 1 and less than or equal to 1.2, it indicates that the eye movement features are almost synchronized with the micro-expression features and speech features, and the corresponding emotions are not realistic; When the value is greater than 1.2 and less than 1.5, it indicates that eye-movement features slightly precede micro-expression features and speech features, and the corresponding emotion may be mixed with controlled behavior; when the leading factor When the value is greater than or equal to 1.5, it indicates that eye movement features significantly lead micro-expression features and speech features, and the corresponding emotions are naturally triggered by internal psychological states to manifest behavior.

7. The emotion recognition method based on multimodal biometrics as described in claim 6, characterized in that: Credibility Index of Genuine Emotions The calculation formula is as follows: ； In the formula: when the credibility index of true emotion is... A higher value indicates that the emotions expressed by the subjects are more authentic and credible. Using eye-tracking feature data as a baseline, the credibility index of authentic emotions is further enhanced through comparative coupling between eye-tracking feature data and micro-expression feature data, and through comparative coupling between eye-tracking feature data and speech feature data. The final judgment is more accurate.

8. A system based on the emotion recognition method based on multimodal biometrics as described in claim 7, characterized in that: It includes a multimodal module (1), a calculation module (2), a dynamic module (3), and a summary module (4); the multimodal module (1) is unidirectionally connected to the calculation module (2); the calculation module (2) is unidirectionally connected to the dynamic module (3) and the summary module (4) respectively; the dynamic module (3) is unidirectionally connected to the summary module (4).

9. A system based on the emotion recognition method based on multimodal biometrics as described in claim 8, characterized in that: The multimodal module (1) includes a test unit (11), an eye-tracking unit (12), a micro-expression unit (13), and a voice unit (14); the test unit (11) is unidirectionally connected to the eye-tracking unit (12), the micro-expression unit (13), and the voice unit (14), respectively; the test unit (11) is also equipped with a microphone, an eye-tracking device, and a face acquisition device; the eye-tracking unit (12) is unidirectionally connected to the calculation module (2); the micro-expression unit (13) is unidirectionally connected to the calculation module (2); and the voice unit (14) is unidirectionally connected to the calculation module (2).

10. A system based on the emotion recognition method based on multimodal biometrics as described in claim 9, characterized in that: The summary module (4) includes a coupling unit (41), a detection unit (42), and a judgment unit (43); the coupling unit (41) and the detection unit (42) are unidirectionally connected; the detection unit (42) and the judgment unit (43) are unidirectionally connected; the dynamic module (3) and the detection unit (42) are unidirectionally connected.

Images