A method, device, system, and storage medium for identifying abnormal emotional responses.

Abstract

This invention discloses a method, apparatus, system, and storage medium for identifying abnormal emotional responses. The method includes: acquiring facial expression image frames of a test subject watching a preset test video, and determining a first emotion based on the facial expression image frames; acquiring the brainwaves of the test subject watching the preset test video, and determining a second emotion based on the brainwaves; determining a first matching degree between the first emotion and the second emotion, and determining whether the test subject's emotional response is abnormal based on the first matching degree and the second matching degree. The technical solution of this invention, by utilizing the matching degree between facial emotions and brain emotions, and the difference between the matching preset emotions and preset emotions in the preset test video, can effectively assist testers in determining whether a test subject's emotional response is abnormal, avoiding the impact of test subjects deliberately suppressing changes in facial expressions on the accuracy of identifying abnormal emotional responses.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interfaces, and more particularly to a method, device, system, and storage medium for identifying abnormal emotional responses. Background Technology

[0002] Emotion recognition can be understood as an individual's recognition of the emotions of others. Currently, it often refers to AI (Artificial Intelligence) automatically identifying an individual's emotional state by acquiring their physiological or non-physiological signals, and is an important component of affective computing. Traditional methods for emotion recognition and monitoring involve having the monitored person watch different types of films, while personnel record and identify the monitored person's facial emotions at any time.

[0003] However, although the monitored subjects can usually watch the test video in its entirety, there are some cases where the monitored subjects' facial expressions do not match their actual emotions, such as the monitored subjects deliberately suppressing changes in their facial expressions, which affects the accuracy of identifying abnormal emotional reactions. Summary of the Invention

[0004] This invention provides a method, device, system, and storage medium for identifying abnormal emotional responses, in order to solve the problem of poor accuracy in judging whether an emotional response is abnormal.

[0005] In a first aspect, the present invention provides a method for identifying abnormal emotional responses, comprising:

[0006] Acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames;

[0007] The brainwaves of the test subject while watching the preset test video were acquired, and the second emotion was determined based on the brainwaves.

[0008] A first matching degree is determined between the first emotion and the second emotion. Based on the first matching degree and the second matching degree, it is determined whether the emotional response of the test subject is abnormal. The preset emotion is a preset emotion that matches the preset test video. The second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes both the first emotion and the second emotion.

[0009] Secondly, the present invention provides a device for identifying abnormal emotional responses, comprising:

[0010] The first emotion determination module is used to acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames.

[0011] The second emotion determination module is used to acquire the brainwaves of the test subject while watching the preset test video, and to determine the second emotion based on the brainwaves.

[0012] An emotion abnormality determination module is used to determine a first matching degree between the first emotion and the second emotion, and to determine whether the emotional reaction of the test subject is abnormal based on the first matching degree and the second matching degree. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes the first emotion and the second emotion.

[0013] Thirdly, the present invention provides an emotion recognition system, the system comprising:

[0014] At least one camera device, at least one brainwave sensor, and at least one processor;

[0015] and memory that is communicatively connected to at least one processor;

[0016] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to perform the method for identifying abnormal emotional responses described in the first aspect.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a processor to execute the above-described method for identifying abnormal emotional responses in the first aspect.

[0018] The present invention provides a scheme for identifying abnormal emotional responses. It acquires facial expression image frames of a test subject watching a preset test video, determines a first emotion based on the facial expression image frames, acquires the test subject's electroencephalogram (EEG) while watching the preset test video, determines a second emotion based on the EEG, determines a first matching degree between the first and second emotions, and determines whether the test subject's emotional response is abnormal based on the first and second matching degrees. The preset emotion is a pre-defined emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes both the first and second emotions. By employing this technical solution, when a test subject watches a preset test video, the facial expression image frames accurately determine the test subject's facial emotion (i.e., the first emotion), and the EEG accurately determines the test subject's brain emotion (i.e., the second emotion). By utilizing the matching degree of these two emotions and the difference between them and the pre-defined emotion matching the preset test video, the tester can effectively assist the tester in determining whether the test subject's emotional response is abnormal, avoiding the impact of the test subject deliberately suppressing facial expressions on the accuracy of identifying abnormal emotional responses.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a method for identifying abnormal emotional responses according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart of a method for identifying abnormal emotional responses according to Embodiment 2 of the present invention;

[0023] Figure 3 This is a flowchart of another method for identifying abnormal emotional responses provided in Embodiment 3 of the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of an emotional reaction abnormality recognition device provided in Embodiment 4 of the present invention;

[0025] Figure 5This is a schematic diagram of the structure of an emotion acquisition device in an emotion recognition system according to Embodiment 5 of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. In the description of this invention, unless otherwise stated, "a plurality of" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0028] Example 1

[0029] Figure 1 The flowchart below shows a method for identifying abnormal emotional reactions according to Embodiment 1 of the present invention. This embodiment is applicable to identifying whether a person's emotional reaction is abnormal. The method can be executed by an abnormal emotional reaction identification device, which can be implemented in hardware and / or software. The abnormal emotional reaction identification device can be configured in an emotion recognition system, which may include at least one camera device, at least one brainwave sensor, at least one processor, and a memory communicatively connected to the processor.

[0030] like Figure 1 As shown, the method for identifying abnormal emotional responses provided in Embodiment 1 of the present invention specifically includes the following steps:

[0031] S101. Obtain facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames.

[0032] In this embodiment, a camera device can be used to acquire images of the test subject watching a preset test video; these images are facial expression frames. Then, using a preset method, such as inputting the image into a preset emotion model, the test subject's facial emotion while watching the preset test video is determined based on the emotion type output by the model—that is, the first emotion. The preset test video can contain multiple segments of test video material of different preset types, such as horror, science fiction, comedy, and sentimental test video material. The duration of each segment can be preset, for example, 10 minutes.

[0033] S102. Obtain the brainwaves of the test subject while watching the preset test video, and determine the second emotion based on the brainwaves.

[0034] In this embodiment, the brainwaves of the test subject while watching the preset test video can be obtained by using a brainwave sensor worn by the test subject. Since the waveform of brainwaves differs under different emotions, the test subject's secondary emotion, also known as brain emotion or psychological emotion, can be determined by analyzing the test subject's brainwaves.

[0035] S103. Determine the first matching degree between the first emotion and the second emotion, and determine whether the emotional reaction of the test subject is abnormal based on the first matching degree and the second matching degree, wherein the preset emotion is a preset emotion that matches the preset test video, the second matching degree is the matching degree between the test emotion and the preset emotion, and the test emotion includes the first emotion and the second emotion.

[0036] In this embodiment, a first emotion and a second emotion are first compared to obtain a first matching degree. The higher the matching degree, the more consistent the test subject's facial and mental emotions are, and the greater the likelihood that the test subject's emotional response is normal. Then, the first and second emotions are compared with preset emotions to obtain a second matching degree. This matching degree includes the first emotion matching degree (i.e., the matching degree between the first emotion and the preset emotion) and the second emotion matching degree (i.e., the matching degree between the second emotion and the preset emotion). The higher the first emotion matching degree, the more consistent the test subject's facial emotions are with the preset emotion, and the greater the likelihood that the facial emotional response is normal. The higher the second emotion matching degree, the more consistent the test subject's mental emotions are with the preset emotion, and the greater the likelihood that the mental emotional response is normal. Therefore, based on the magnitude of the first and second matching degrees, it can be determined whether the test subject's emotional response is abnormal. Here, the preset emotion can be understood as the emotion corresponding to a preset test film, such as fear corresponding to a horror-themed preset test film.

[0037] Optionally, a monitoring person can be pre-determined to observe the test subject's emotions during the playback of a preset test video. This ensures that the progress of the abnormal emotion response recognition test can be adjusted in real time, such as switching to a different test video before the current video ends, thus guaranteeing the effectiveness and efficiency of the abnormal emotion response recognition test.

[0038] The method for identifying abnormal emotional responses provided in this invention involves acquiring facial expression image frames of a test subject watching a preset test video, determining a first emotion based on the facial expression image frames, acquiring the test subject's electroencephalogram (EEG) while watching the preset test video, determining a second emotion based on the EEG, determining a first matching degree between the first and second emotions, and determining whether the test subject's emotional response is abnormal based on the first and second matching degrees. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes both the first and second emotions. This invention allows for accurate determination of the test subject's facial emotion (i.e., the first emotion) and brain emotion (i.e., the second emotion) based on the facial expression image frames while watching the preset test video. By utilizing the matching degree of these two emotions and the difference between them and the preset emotion matching the preset test video, the method effectively assists testers in determining whether the test subject's emotional response is abnormal, avoiding the impact of the test subject deliberately suppressing facial expressions on the accuracy of identifying abnormal emotional responses.

[0039] Example 2

[0040] Figure 2 This is a flowchart of a method for identifying abnormal emotional reactions provided in Embodiment 2 of the present invention. The technical solution of the present invention is further optimized based on the above optional technical solutions, and provides a specific way to identify whether a person's emotional reaction is abnormal.

[0041] Optionally, before determining the first emotion based on the facial expression image frame, the method further includes: determining virtual positional displacement change information based on the dense ultrasonic feedback points on the test subject's face while watching a preset test video, wherein the dense ultrasonic feedback points are the points reflected back after being emitted to the face, and the virtual positional displacement change information is used to characterize the changes in the test subject's facial expression; wherein determining the first emotion based on the facial expression image frame includes: determining the correspondence between the dense ultrasonic feedback points and the pixels in the facial expression image frame; and determining the test subject's first emotion based on the correspondence and the virtual positional displacement change information. The advantage of this approach is that by determining the correspondence between the dense ultrasonic feedback points and the pixels in the facial expression image frame, it is more conducive to accurately analyzing facial expression changes under real facial changes, thereby improving the accuracy of identifying abnormal emotional responses.

[0042] Optionally, determining whether the test subject's emotional response is abnormal based on the first matching degree and the second matching degree includes: if the first matching degree is greater than a first preset threshold and the second matching degree is less than or equal to a second preset threshold, then the test subject's emotional response is determined to be abnormal and the abnormality level is level one; if the first matching degree is less than or equal to the first preset threshold and the second matching degree is less than or equal to the second preset threshold, then the emotional response is determined to be abnormal and the abnormality level is level two, wherein the degree of abnormality in level two is deeper than that in level one. The advantage of this setting is that, by utilizing predetermined thresholds, test subjects with abnormal emotional responses, such as inconsistencies between facial and brain emotions, and inconsistencies between facial and / or brain emotions and normal emotions, can be accurately identified.

[0043] like Figure 2 As shown in Embodiment 2 of the present invention, a method for identifying abnormal emotional responses specifically includes the following steps:

[0044] S201. Obtain facial expression image frames of the test subject while watching the preset test video, and determine the virtual position displacement change information based on the dense ultrasonic feedback array points on the test subject's face while watching the preset test video.

[0045] Among them, the dense array of ultrasonic feedback points are the points that emit dense arrays of ultrasonic waves to the face and then reflect them back. The virtual position displacement change information is used to characterize the changes in the facial expressions of the test subject.

[0046] Specifically, an ultrasonic probe can be used to emit a dense array of ultrasonic waves to the face of the person being tested. The reflected points, i.e., the dense array of ultrasonic feedback points, will be reflected back to the ultrasonic probe. As facial expressions change, the position information of the dense array of ultrasonic feedback points reflected on the ultrasonic probe will also change. The change information corresponding to this change is the virtual position displacement change information.

[0047] S202. Determine the correspondence between the densely packed ultrasonic feedback array points and the pixels in the facial expression image frame.

[0048] Specifically, a facial expression image frame contains multiple facial pixels, and the correspondence between each point in the dense ultrasonic feedback array and each facial pixel can be determined. The specific method for determining the correspondence is not limited here.

[0049] S203. Determine the test subject's first emotion based on the correspondence and virtual position displacement information.

[0050] Specifically, based on the aforementioned correspondence and virtual position displacement change information, the real-time determined virtual position displacement change information can be calibrated and mapped to frame points (i.e., facial pixels) in the real-time captured facial expression image frames. This allows us to determine how the test subject's face changes, and based on these changes, the test subject's primary emotion can be determined. For example, the facial change features of the test subject can be input into a preset facial emotion recognition model, and the primary emotion can be determined based on the emotion result output by the model.

[0051] Optionally, the test subject's first emotion can be determined based on the correspondence and virtual position displacement change information. This includes: mapping the virtual position displacement change information onto facial expression image frames according to the correspondence to obtain the test subject's facial change information; and comparing the facial change information with preset facial model data to obtain the test subject's first emotion. The advantage of this setting is that by comparing the facial change information with preset facial model data, the test subject's first emotion can be accurately and quickly determined.

[0052] Specifically, as described above, after calibrating the real-time determined virtual positional displacement change information to the frame points (i.e., facial pixels) in the real-time captured facial expression image frames, the facial change information of the test subject can be obtained. Then, this facial change information is compared with each facial pattern in the preset facial pattern data. The emotion corresponding to the facial pattern data that is closest to this facial change information is determined as the first emotion. For example, if the facial change information is that position A of the face is shifted upwards by a millimeter, and facial pattern data 1 in the preset facial pattern data is also that position A of the face is shifted upwards by a millimeter, then the first emotion of the test subject can be determined as the emotion corresponding to facial pattern data 1. The preset facial pattern data contains multiple preset facial pattern data, and each facial pattern data corresponds to a different emotion.

[0053] S204. Obtain the brainwaves of the test subject while watching a preset test video, and determine the second emotion based on the brainwaves.

[0054] S205. Determine the first matching degree of the first emotion and the second emotion, and determine whether the first matching degree is greater than the first preset threshold and the second matching degree is less than or equal to the second preset threshold. If yes, proceed to step 206; otherwise, proceed to step 207.

[0055] Specifically, when the first matching degree is greater than the first preset threshold, it can be determined that the first emotion and the second emotion are consistent, indicating that the test subject's brain emotion and facial emotion are consistent. When the second matching degree is less than or equal to the second preset threshold, it can be determined that either the first emotion matching degree or the second emotion matching degree is less than or equal to the second preset threshold. When the first emotion matching degree is less than or equal to the second preset threshold, it indicates that the test subject's brain emotion is inconsistent with the emotion expected when watching the preset test film. When the second emotion matching degree is less than or equal to the second preset threshold, it indicates that the test subject's facial emotion is inconsistent with the emotion expected when watching the preset test film.

[0056] S206. Determine that the test subject's emotional reaction is abnormal and the abnormality level is the first level.

[0057] Specifically, as shown above, if so, it can be determined that the test subject's emotional reaction is abnormal, and the level is the first level.

[0058] S207. Determine whether the first matching degree is less than or equal to the first preset threshold and the second matching degree is less than or equal to the second preset threshold. If yes, proceed to step 208; otherwise, end.

[0059] Specifically, when the first matching degree is less than or equal to the first preset threshold, it can be determined that the first emotion and the second emotion are inconsistent, which means that the test subject's brain emotion and facial emotion are inconsistent.

[0060] S208. Determine that the emotional reaction is abnormal and the abnormality level is the second level.

[0061] The second level of anomaly is more severe than the first level.

[0062] Specifically, since the test subject's brain emotions and facial emotions are inconsistent, and at least one of the brain emotions and facial emotions is inconsistent with the emotions that should be present when watching the preset test video, it can be determined that the test subject's emotional reaction is abnormal, and the level is the second level, which is more abnormal than the first level.

[0063] Optionally, when the first matching degree is greater than the first preset threshold and the second matching degree is greater than the second preset threshold, the test subject's emotional response can be determined to be normal; when the first matching degree is less than or equal to the first preset threshold and the second matching degree is greater than the second preset threshold, the test subject's emotional response can be determined to be abnormal, and the abnormality level is level five. The abnormality level two is more severe than level five.

[0064] The method for identifying abnormal emotional responses provided in this invention determines the correspondence between densely packed ultrasonic feedback array points and pixels in facial expression image frames, which is more conducive to accurately analyzing facial expression changes under real facial changes, thereby improving the accuracy of identifying abnormal emotional responses. Furthermore, by using a pre-determined threshold, it can accurately identify test subjects with abnormal emotional responses, such as those whose facial emotions and / or brain emotions are inconsistent with normal emotions. This avoids the impact on the accuracy of identifying abnormal emotional responses caused by test subjects deliberately suppressing their facial expressions.

[0065] Example 3

[0066] Figure 3 This is a flowchart of another method for identifying abnormal emotional reactions provided in Embodiment 3 of the present invention. The technical solution of the present invention is further optimized based on the above optional technical solutions, and provides a specific way to identify whether a person's emotional reaction is abnormal.

[0067] Optionally, before determining whether the test subject's emotional response is abnormal based on the first and second matching degrees, the method further includes: determining the electroencephalogram (EEG) power spectrum ratio based on the beta rhythm wave signal in the test subject's frontal region and the theta rhythm wave signal in the temporal lobe region, and determining the test subject's attention index based on the EEG power spectrum ratio; determining the degree of relaxation based on the alpha rhythm wave signal in the test subject's frontal region, and determining the test subject's sluggishness time based on the facial expression image frame and the EEG, wherein the sluggishness time is the time difference between the test subject's facial expression change and brain emotion change; wherein determining whether the test subject's emotional response is abnormal based on the first and second matching degrees includes: determining whether the test subject's emotional response is abnormal based on physiological information, the first matching degree, and the second matching degree, wherein the physiological information includes the attention index, the degree of relaxation, and the sluggishness time. The advantage of this setup is that the attention index can be used to determine whether the test subject's attention is abnormal, the relaxation level can be used to determine whether the test subject is overly tense, and the lag time can be used to determine whether the test subject's reaction ability is abnormal. By comprehensively combining the above physiological information, the first matching degree, and the second matching degree, it is possible to comprehensively determine whether the test subject's emotional reaction is abnormal.

[0068] like Figure 3 As shown, another method for identifying abnormal emotional responses provided in Embodiment 3 of the present invention specifically includes the following steps:

[0069] S301. Acquire facial expression image frames of the test subject while watching the preset test video, and determine the virtual position displacement change information based on the dense ultrasonic feedback array points on the test subject's face while watching the preset test video.

[0070] S302. Determine the correspondence between the densely packed ultrasonic feedback array points and the pixels in the facial expression image frame.

[0071] S303. Determine the test subject's first emotion based on the correspondence and virtual position displacement information.

[0072] S304. Obtain the brainwaves of the test subject while watching a preset test video, and determine the second emotion based on the brainwaves.

[0073] S305. Determine the first match degree between the first emotion and the second emotion. Based on the β rhythm wave signal in the frontal region and the θ rhythm wave signal in the temporal lobe region of the test subject, determine the EEG power spectrum ratio, and determine the test subject's attention index based on the EEG power spectrum ratio.

[0074] Specifically, brainwave sensors can be used to acquire the electroencephalogram (EEG) signals of the test subject. When a person is under mental stress, the rhythmic wave signals in the frontal region are more obvious. When a person is fatigued or drowsy, the beta rhythmic waves in the temporal lobe of the brain are more obvious. Since the intensity of EEG signals is at the microvolt level, it is extremely weak and easily affected by 50 Hz AC power and polarization levels below 0.5 Hz. To ensure the accuracy of the results, the EEG signals can be denoised twice. FIR (Finite Impulse Response) digital filtering is used to filter out signals from 1 to 40 Hz. Then, a moving window value filtering method is used to correct baseline drift, and wavelet coefficient thresholding is used to remove artifacts such as electrooculography (EOG) and electromyography (EMG) in the signal. In the Beta state (a state of alertness and tension), the brainwave signal is 12 to 16 Hz; in the Alpha state (a state of relaxation and detachment from stress), the brainwave signal is 8 to 12 Hz; in the Theta state (a state of reduced or no awareness of bodily states), the brainwave signal is 4 to 8 Hz; and in the Delta state (a state of sleep), the brainwave signal is 0.5 to 4 Hz.

[0075] For example, the EEG power spectrum ratio R can be determined as: R = (α + β) / θ, where α is the α-rhythmic wave signal in the prefrontal cortex, β is the β-rhythmic wave signal in the prefrontal cortex, and θ is the θ-rhythmic wave signal in the temporal lobe. The attention index A can be determined as: A = X - (RZ) * Y, where X, Y, and Z are coefficients. X and Y can both be 100 or different from each other, and Z can be 0.95. Typically, when attention is highly concentrated, the attention index is between 60 and 80; when attention is relatively concentrated, the attention index is between 40 and 60; and when attention is relatively scattered, the attention index is between 20 and 40. If the attention index is below 20, it indicates severely scattered attention.

[0076] S306. Determine the degree of relaxation based on the alpha rhythm wave signal in the frontal region of the test subject, and determine the sluggishness time of the test subject based on facial expression image frames and EEG.

[0077] The sluggishness time is the time difference between changes in facial expression and changes in emotional state in the brain of the test subject.

[0078] For example, the relaxation level M can be determined as: M = α / M*N, where M and N are coefficients, M can be 10, and N can be 100. The sluggishness time can be determined as the difference between the time T1 of the first emotion determined from facial expression image frames and the time T2 of the second emotion determined from EEG.

[0079] S307. Based on physiological information, first matching degree, and second matching degree, determine whether the test subject's emotional response is abnormal.

[0080] The physiological information includes attention index, relaxation level, and sluggishness time.

[0081] Specifically, since physiological information can indicate whether the test subject has any abnormalities to a certain extent, such as when any item in the physiological information is too large or too small, it can be determined that the test subject may have problems with emotional response due to any of the factors such as attention, relaxation, and reaction time. Therefore, in addition to determining whether the test subject's emotional response is abnormal based on the first and second matching degrees, physiological information can also be added as a basis for judgment.

[0082] Optionally, based on physiological information, a first matching degree, and a second matching degree, it can be determined whether the test subject's emotional response is abnormal. This includes: if the first matching degree is less than or equal to a first preset threshold, any item in the physiological information is less than or equal to its corresponding preset threshold, and the second matching degree is less than or equal to a second preset threshold, then the test subject's emotional response is determined to be at level three abnormality. The advantage of this setting is that by determining the relationship between any item in the physiological information and its corresponding preset threshold, test subjects with abnormal emotional responses who have problems with any of the following: attention, relaxation, or responsiveness.

[0083] Specifically, when the first matching degree is less than or equal to the first preset threshold, any item in the physiological information is less than or equal to the corresponding preset threshold, and the second matching degree is less than or equal to the second preset threshold, it indicates that the test subject has any of the following problems: abnormal attention, abnormal relaxation, or slow reaction. Additionally, there is inconsistency between facial expressions and the test subject's emotional state, and at least one of the facial expressions is inconsistent with the emotions expected when watching the preset test video. Therefore, the test subject's emotional response can be determined to be abnormal, and the level is classified as Level 3. The degree of abnormality of Level 3 is not specified in relation to Levels 1 and 2.

[0084] Optionally, based on physiological information, a first matching degree, and a second matching degree, it can be determined whether the test subject's emotional response is abnormal. This includes: if the first matching degree is less than or equal to a first preset threshold, each item in the physiological information is less than or equal to its corresponding preset threshold, and the second matching degree is less than or equal to a second preset threshold, then the test subject's emotional response is determined to be abnormal, and the abnormality level is level four. The abnormality level four is more severe than level three. The advantage of this setting is that it can accurately screen out test subjects with abnormal emotional responses who have problems with attention, relaxation, and responsiveness.

[0085] Specifically, when the first matching degree is less than or equal to the first preset threshold, each item in the physiological information is less than or equal to its corresponding preset threshold, and the second matching degree is less than or equal to the second preset threshold, it indicates that the test subject has problems with attention abnormalities, relaxation abnormalities, and slow reactions, as well as inconsistencies between facial emotions and other facial expressions, and at least one of the facial emotions is inconsistent with the emotions expected when watching the preset test video. Therefore, it can be determined that the test subject's emotional response is abnormal, and the level is fourth. Since there are more problems corresponding to the fourth level than to the third level, the degree of abnormality in the fourth level is more severe than that in the third level.

[0086] The method for identifying abnormal emotional responses provided in this invention uses an attention index to determine whether the test subject's attention is abnormal, a relaxation level to determine whether the test subject is overly tense, and a lag time to determine whether the test subject's reaction ability is abnormal. By comprehensively combining the above physiological information, the first matching degree, and the second matching degree, the method can comprehensively determine whether the test subject's emotional response is abnormal, avoiding the impact on the accuracy of identifying abnormal emotional responses due to the test subject deliberately suppressing changes in facial expressions.

[0087] Example 4

[0088] Figure 4 This is a schematic diagram of the structure of a device for recognizing abnormal emotional responses provided in Embodiment 4 of the present invention. Figure 4 As shown, the device includes: a first emotion determination module 401, a second emotion determination module 402, and an emotion abnormality determination module 403, wherein:

[0089] The first emotion determination module is used to acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames.

[0090] The second emotion determination module is used to acquire the brainwaves of the test subject while watching the preset test video, and to determine the second emotion based on the brainwaves.

[0091] An emotion abnormality determination module is used to determine a first matching degree between the first emotion and the second emotion, and to determine whether the emotional reaction of the test subject is abnormal based on the first matching degree and the second matching degree. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes the first emotion and the second emotion.

[0092] The emotional reaction abnormality identification device provided in this invention allows for the accurate determination of the test subject's facial emotions (i.e., the first emotion) based on facial expression image frames and the accurate determination of the test subject's brain emotions (i.e., the second emotion) based on electroencephalograms (EEGs). By utilizing the matching degree between these two emotions and the differences between them and the preset emotions matched in the preset test video, the device can effectively assist testers in determining whether the test subject's emotional reaction is abnormal. This avoids the accuracy of emotional reaction abnormality identification being affected by factors such as the test subject deliberately suppressing changes in their facial expressions.

[0093] Optionally, the device may also include:

[0094] The displacement change information determination module is used to determine virtual position displacement change information based on the dense ultrasonic feedback array points on the face of the test subject when watching a preset test film, before determining the first emotion based on the facial expression image frame. The dense ultrasonic feedback array points are the array points reflected back after being emitted to the face. The virtual position displacement change information is used to characterize the changes in the facial expression of the test subject.

[0095] Optionally, the first emotion determination module includes:

[0096] The correspondence determination unit is used to determine the correspondence between the densely packed ultrasonic feedback array points and the pixels in the facial expression image frame;

[0097] The first emotion determination unit is used to determine the first emotion of the test subject based on the correspondence and the virtual position displacement change information.

[0098] Optionally, determining the first emotion of the test subject based on the correspondence and the virtual position displacement change information includes: mapping the virtual position displacement change information to the facial expression image frame according to the correspondence to obtain the facial change information of the test subject; and comparing the facial change information with preset facial makeup data to obtain the first emotion of the test subject.

[0099] Optional, the emotion abnormality detection module includes:

[0100] The first level determination unit is used to determine that the emotional reaction of the test subject is abnormal and the abnormality level is the first level if the first matching degree is greater than the first preset threshold and the second matching degree is less than or equal to the second preset threshold.

[0101] The second-level determination unit is used to determine that the emotional reaction is abnormal and the abnormality level is second level if the first matching degree is less than or equal to the first preset threshold and the second matching degree is less than or equal to the second preset threshold, wherein the degree of abnormality of the second level is deeper than that of the first level.

[0102] Optionally, the device may also include:

[0103] The ratio and index determination module is used to determine the EEG power spectrum ratio based on the β rhythm wave signal of the frontal region and the θ rhythm wave signal of the temporal lobe region of the test subject before determining whether the emotional response of the test subject is abnormal based on the first matching degree and the second matching degree, and to determine the attention index of the test subject based on the EEG power spectrum ratio.

[0104] The sluggishness time determination module is used to determine the degree of relaxation based on the alpha rhythm wave signal of the frontal region of the test subject, and to determine the sluggishness time of the test subject based on the facial expression image frame and the electroencephalogram, wherein the sluggishness time is the time difference between the change in the test subject's facial expression and the change in the brain's emotional state.

[0105] Optional, the emotion abnormality detection module includes:

[0106] An emotion abnormality determination unit is used to determine whether the emotional response of the test subject is abnormal based on physiological information, the first matching degree, and the second matching degree, wherein the physiological information includes the attention index, the relaxation degree, and the sluggishness time.

[0107] Optionally, determining whether the test subject's emotional response is abnormal based on physiological information, the first matching degree, and the second matching degree includes: if the first matching degree is less than or equal to a first preset threshold, any item in the physiological information is less than or equal to the corresponding preset threshold, and the second matching degree is less than or equal to a second preset threshold, then the test subject's emotional response is determined to be at level three abnormality.

[0108] Optionally, determining whether the test subject's emotional response is abnormal based on physiological information, the first matching degree, and the second matching degree includes: if the first matching degree is less than or equal to a first preset threshold, each item in the physiological information is less than or equal to the corresponding preset threshold, and the second matching degree is less than or equal to the second preset threshold, then the test subject's emotional response is determined to be abnormal and the abnormality level is level four, wherein the abnormality level of level four is more severe than that of level three.

[0109] The emotional reaction abnormality identification device provided in the embodiments of the present invention can execute the emotional reaction abnormality identification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0110] Example 5

[0111] Figure 5 A schematic diagram of an emotion acquisition device in an emotion recognition system that can be used to implement embodiments of the present invention is shown. Figure 5 As shown, the emotion acquisition device may include a housing 1, a camera 2 (i.e., a camera device in an emotion recognition system), an ultrasonic probe 3, a recognition development board 4, a brainwave sensor 5, a power supply 6, a first connecting frame 7, a first rotating post 8, a first stud 9, a pressure ring 10, a first washer 11, a first nut 12, a flip rod 13, a second rotating post 14, a second connecting frame 15, a second stud 16, a second pressure ring 17, a second washer 18, a second nut 19, and a mounting plate 20.

[0112] Cameras 2 are embedded on all four sides of the left side of the outer casing 1, allowing for the capture of facial expression image frames from four directions, thus avoiding any missing frames. An ultrasonic probe 3 is embedded in the center of one side of the outer casing 1. The ultrasonic probe 3 emits a dense array of ultrasonic waves to the face of the test subject, and the ultrasonic feedback points are reflected back into the ultrasonic probe 3, which can quickly capture changes in the virtual positional displacement of the face to obtain virtual positional displacement information. Inside the outer casing 1 is a recognition development board 4 that receives facial expression image frames and virtual positional displacement information for emotion recognition and monitoring. The EEG input terminal of the recognition development board 4 is connected to the transmission line of the brainwave sensor 5. A power supply 6 is located on the outer casing 1. The power output terminal of the power supply 6 is connected to the power input terminal of the recognition development board 4 via a cable, and the main power line of the power supply 6 is connected to an external power supply. The recognition development board 4 includes features for real-time reception of facial expression image frames captured by the cameras 2, real-time reception of virtual positional displacement information captured by the ultrasonic probe, and real-time reception of EEG signals uploaded by the brainwave sensor 5.

[0113] A flip adjustment device is installed on the right side of the outer casing 1. The flip adjustment device includes a first connecting frame 7 welded to the outer casing 1. The first connecting frame 7 is a U-shaped frame, and a first rotating post 8 is inserted into the first connecting frame 7. A first stud 9 is welded to the center of both ends of the first rotating post 8. A pressure ring 10 is fitted on each of the first studs 9 and slides out of the first connecting frame 7. A first washer 11 is fitted on the end of the first stud 9 that extends out of the first connecting frame 7 and is locked by a first nut 12. When the first nut 12 locks the first stud 9, the first washer 11 presses against the first connecting frame 7. The pressure ring 10 presses against the first rotating post 8, stabilizing the rotational position of the first rotating post 8 in the first connecting frame 7. A flip rod 13 is welded to the first rotating post 8, and a second rotating post 14 is welded to the upper end of the flip rod 13. A second connecting frame 15 is fitted on the second rotating post 14, and a second stud 16 is welded to the center of both ends of the second rotating post 14. Each second stud 16 is fitted with a pressure ring 17, which slides out of the second connecting frame 15. Each end of the second stud 16 extending out of the second connecting frame 15 is fitted with a second washer 18 and locked by a second nut 19. The second washer 18 presses against the second connecting frame 15. The pressure ring 17 presses against the second rotating post 14, stabilizing its rotational position within the second connecting frame 15. A mounting plate 20 is integrally formed on the second connecting frame 15, and the mounting plate 20 has mounting holes for installing expansion bolts.

[0114] The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0115] The emotion recognition system also includes at least one processor and a memory, such as a read-only memory (ROM) or a random access memory (RAM), which is communicatively connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor. The processor can perform various appropriate actions and processes based on the computer program stored in the read-only memory (ROM) or the computer program loaded from the storage unit into the random access memory (RAM).

[0116] In some embodiments, the method for identifying abnormal emotional responses can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the method for identifying abnormal emotional responses described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the method for identifying abnormal emotional responses by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] The computer equipment provided above can be used to execute the method for identifying abnormal emotional reactions provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0120] Example 6

[0121] In the context of this invention, the computer-readable storage medium may be a tangible medium, and the computer-executable instructions, when executed by a computer processor, are used to perform a method for recognizing abnormal emotional responses, the method comprising:

[0122] Acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames;

[0123] The brainwaves of the test subject while watching the preset test video were acquired, and the second emotion was determined based on the brainwaves.

[0124] A first matching degree is determined between the first emotion and the second emotion, and based on the first matching degree and the second matching degree, it is determined whether the emotional response of the test subject is abnormal. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes the first emotion and the second emotion.

[0125] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by, or in conjunction with, an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0126] The computer equipment provided above can be used to execute the method for identifying abnormal emotional reactions provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0127] It is worth noting that in the embodiments of the above-mentioned abnormal emotional response recognition device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0128] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for identifying abnormal emotional responses, characterized in that, include: Acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames; The brainwaves of the test subject while watching the preset test video were acquired, and the second emotion was determined based on the brainwaves. A first matching degree is determined between the first emotion and the second emotion, and based on the first matching degree and the second matching degree, it is determined whether the emotional response of the test subject is abnormal. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes the first emotion and the second emotion. Prior to determining the first emotion based on the facial expression image frame, the method further includes: Based on the dense array of ultrasonic feedback points on the face of the test subject while watching a preset test video, virtual position displacement change information is determined. The dense array of ultrasonic feedback points are the points reflected back after being emitted to the face. The virtual position displacement change information is used to characterize the changes in the test subject's facial expressions. The step of determining the first emotion based on the facial expression image frame includes: Determine the correspondence between the densely packed ultrasonic feedback array points and the pixels in the facial expression image frame; Based on the correspondence and the virtual position displacement change information, the first emotion of the test subject is determined.

2. The method according to claim 1, characterized in that, Determining the test subject's first emotion based on the correspondence and the virtual position shift information includes: Based on the correspondence, the virtual position displacement change information is mapped to the facial expression image frame to obtain the facial change information of the test subject; The facial change information is compared with preset facial data to obtain the test subject's first emotion.

3. The method according to claim 1, characterized in that, The step of determining whether the test subject's emotional response is abnormal based on the first matching degree and the second matching degree includes: If the first matching degree is greater than the first preset threshold and the second matching degree is less than or equal to the second preset threshold, then the emotional reaction of the test subject is determined to be abnormal and the abnormality level is the first level. If the first matching degree is less than or equal to the first preset threshold and the second matching degree is less than or equal to the second preset threshold, then the emotional reaction is determined to be abnormal and the abnormality level is the second level, wherein the degree of abnormality of the second level is deeper than that of the first level.

4. The method according to claim 1, characterized in that, Before determining whether the test subject's emotional response is abnormal based on the first and second matching scores, the method further includes: The EEG power spectrum ratio is determined based on the β rhythm wave signal in the frontal region and the θ rhythm wave signal in the temporal lobe region of the test subject, and the attention index of the test subject is determined based on the EEG power spectrum ratio. The degree of relaxation is determined based on the alpha rhythm wave signal in the forehead region of the test subject, and the sluggishness time of the test subject is determined based on the facial expression image frame and the electroencephalogram, wherein the sluggishness time is the time difference between the change in facial expression and the change in brain emotion of the test subject. The step of determining whether the test subject's emotional response is abnormal based on the first matching degree and the second matching degree includes: Based on physiological information, the first matching degree, and the second matching degree, it is determined whether the emotional response of the test subject is abnormal, wherein the physiological information includes the attention index, the relaxation degree, and the sluggishness time.

5. The method according to claim 4, characterized in that, The step of determining whether the test subject's emotional response is abnormal based on physiological information, the first matching degree, and the second matching degree includes: If the first matching degree is less than or equal to the first preset threshold, any item in the physiological information is less than or equal to the corresponding preset threshold, and the second matching degree is less than or equal to the second preset threshold, then the emotional response of the test subject is determined to be a level three abnormality.

6. The method according to claim 5, characterized in that, The step of determining whether the test subject's emotional response is abnormal based on physiological information, the first matching degree, and the second matching degree includes: If the first matching degree is less than or equal to the first preset threshold, each item in the physiological information is less than or equal to the corresponding preset threshold, and the second matching degree is less than or equal to the second preset threshold, then the emotional reaction of the test subject is determined to be abnormal and the abnormality level is the fourth level, wherein the abnormality level of the fourth level is deeper than that of the third level.

7. A device for identifying abnormal emotional responses, characterized in that, include: The first emotion determination module is used to acquire facial expression image frames of the test subject while watching a preset test video, and determine the first emotion based on the facial expression image frames. The second emotion determination module is used to acquire the brainwaves of the test subject while watching the preset test video, and to determine the second emotion based on the brainwaves. An emotion abnormality determination module is used to determine the first matching degree between the first emotion and the second emotion, and to determine whether the emotional reaction of the test subject is abnormal based on the first matching degree and the second matching degree. The preset emotion is a preset emotion that matches the preset test video, and the second matching degree is the matching degree between the test emotion and the preset emotion. The test emotion includes the first emotion and the second emotion. The displacement change information determination module is used to determine virtual position displacement change information based on the dense ultrasonic feedback array points on the face of the test subject when watching a preset test film before determining the first emotion based on the facial expression image frame. The dense ultrasonic feedback array points are the array points reflected back after being emitted to the face. The virtual position displacement change information is used to characterize the changes in the facial expression of the test subject. The first emotion determination module includes: The correspondence determination unit determines the correspondence between the densely packed ultrasonic feedback array points and the pixels in the facial expression image frame; The first emotion determination unit is used to determine the first emotion of the test subject based on the correspondence and the virtual position displacement change information.

8. An emotion recognition system, characterized in that, The system includes: At least one camera device, at least one brainwave sensor, at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for identifying abnormal emotional responses as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method for identifying abnormal emotional responses as described in any one of claims 1-6.

Images