Emotion detection method and system based on electroencephalogram signal, face data and eye movement tracking technology fusion and information pushing method

Abstract

The invention discloses an emotion detection method and system based on electroencephalogram signal, face data and eye movement tracking technology fusion and an information pushing method.The detection method comprises the steps that electroencephalogram data, face data and eye movement data of a testee are collected, and weight operation is conducted on the electroencephalogram data, the face data and the eye movement data to output emotion indexes; the emotion state of the testee is judged in an auxiliary mode according to the emotion index; comprehensive operation is carried out by combining the electroencephalogram data, the face data and the eye movement data, so that emotion detection of the user is more efficient and accurate.

Description

Technical Field

[0001] The present invention relates to the field of biological detection, and in particular to an emotion detection method, system and information push method based on the fusion of electroencephalogram signals, facial data and eye tracking technology. Background Art

[0002] Emotion detection, also known as emotion recognition, refers to inferring an individual's current emotional state by analyzing and processing signals collected from sensors. This process involves the detection of physiological signals (such as breathing, heart rate and body temperature) and emotional behaviors (such as facial expressions, voice status and gestures). Generally, the device is used to collect the EEG signals of the subject, and the specific features in the EEG (Electroencephalogram) signal are analyzed based on the EEG signals, such as the power spectrum density of different frequency bands, event-related potentials, etc., to identify the individual's emotional state. EEG signals can be divided into five frequency bands according to frequency characteristics, namely Delta (0.5~4Hz), Theta (4~8Hz), Alpha (8~13Hz), Beta (13~30Hz) and Gamma (30~47Hz) bands, each of which represents different brain activity areas and characteristic states. Emotional values ​​are divided by the numerical values ​​of emotional analysis feature states. However, EEG signals are easily affected by various artifacts during the measurement process. These artifacts will reduce the accuracy of EGG signals, such as electromyographic interference, electrocardiographic interference, 50Hz interference, and some high-frequency signal interference. Therefore, there are certain detection limitations. In addition, different individuals have different physiological characteristics such as scalp thickness and resistivity, which will affect the recording quality of EEG signals. It can be seen that a solution that can effectively and accurately detect the emotions of the subjects is urgently needed by society. Summary of the invention

[0003] The present invention provides a solution to the technical problem that emotion detection in the prior art relies solely on electroencephalogram signals.

[0004] To achieve the above object, the present invention provides an emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology, comprising the following steps: Collecting EEG data, facial data and eye movement data of the subject, and performing weight calculation on the EEG data, the facial data and the eye movement data to output an emotion index; The emotional index is used to assist in judging the emotional state of the subject.

[0005] As an improved solution of the present application, the steps of collecting EEG data are as follows: Media resources with different emotional contents are played to the subjects to induce emotional data with different emotional characteristics, and the EEG data can be obtained by recording the EEG value of each emotional data.

[0006] As an improved solution of the present application, the eye movement data is collected as follows: Media resources with different emotional contents are played to the subjects to record the subjects' eye movement trajectories, and their attention and interest points under the media resources with different emotional contents are analyzed.

[0007] As an improved solution of the present application, the steps of collecting facial data are as follows: Media resources with different emotional contents are played for the subjects to watch so as to capture the facial expressions of the subjects, so that the facial expressions are matched with the facial library data and then the facial data is output.

[0008] As an improved solution of the present application, the media resource includes video, audio or text.

[0009] As an improved solution of the present application, the emotional content includes fear content, anger content, sadness content and happiness content.

[0010] As an improved solution of the present application, the weight calculation specifically includes: The weight calculation specifically includes: S1: Make a preliminary guess about the subject’s emotions based on facial data and obtain a preliminary judgment of the emotions; S2: Matching the initially judged emotion with the corresponding emotion in the EEG data to obtain a first judgment auxiliary value; S3: Matching the preliminary judged emotion with the corresponding emotion in the eye movement data to obtain a second judgement auxiliary value; S4: Obtain the highest value of the gradient standard value, perform weighted operation on the preliminary judgment emotion, the first judgment auxiliary value, and the second judgment auxiliary value to obtain an operation result, compare the operation structure with the standard value, and if the operation result is greater than the standard value, use the preliminary judgment value as the output emotion value and the trustworthiness of the emotion value; S5: If it is less than the operation structure, the secondary value of the gradient standard value is taken as the highest value, and the process loops into S4 and S5 until the sentiment value and the confidence of the sentiment value are output.

[0011] The present application also provides an emotion detection system based on the fusion of EEG signals, facial data and eye tracking technology, including an electronic headband, a camera and a processor; The electronic headband collects EEG data of the subject; The camera collects facial data and eye movement data; The processor performs weighting operations on the EEG data, the facial data, and the eye movement data to output an emotion index, and assists in determining the emotional state of the subject based on the emotion index.

[0012] The present application also provides an information inference method for emotion detection based on the fusion of EEG signals, facial data and eye tracking technology, comprising the following steps: Collecting EEG data, facial data and eye movement data of the subject, and performing weight calculation on the EEG data, the facial data and the eye movement data to output an emotion index; Assist in judging the emotional state of the subject based on the emotional index; The emotional state is collected and associated with a user history preference database and a content tag database to generate push information.

[0013] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention provides an emotion detection method, system and information push method based on the fusion of EEG signals, facial data and eye movement tracking technology. The detection method includes: collecting the EEG data, facial data and eye movement data of the subject, performing weighted operations on the EEG data, facial data and eye movement data to output an emotion index; assisting in judging the emotional state of the subject based on the emotion index; and performing comprehensive operations by combining the EEG data, facial data and eye movement data to make the emotion detection of the user more efficient and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a system schematic diagram of the present invention; Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0015] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with the accompanying drawings.

[0016] In the following description, the example details are given to provide a deeper understanding of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. It should be understood that the specific embodiments are only used to explain the present invention and are not used to limit the present invention.

[0017] It should be understood that when the terms "include" and / or "comprises" are used in this specification, they indicate the existence of the stated features, integers, steps, operations, elements or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components or combinations thereof.

[0018] In order to solve the above technical problems, this application provides an emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology. Figure 2 , including the following steps: Collecting EEG data, facial data and eye movement data of the subject, and performing weight calculation on the EEG data, the facial data and the eye movement data to output an emotion index; Assist in judging the emotional state of the subject based on the emotional index. ; By combining EEG data, facial data and eye movement data for comprehensive calculations, user emotion detection becomes more efficient and accurate.

[0019] As for the collection of EEG data, the steps of collecting EEG data are as follows: play media resources with different emotional contents for the subjects to watch so as to induce emotional data with different emotional characteristics, and record the EEG values ​​of each emotional data to obtain the EEG data; the EEG data are EEG values ​​associated with time parameters. Specifically, different emotional contents are content items, and the subjects' stay time and EEG values ​​in the corresponding content items are recorded. The frequency bands of the EEG values ​​of any content item are Delta (0.5~4Hz), Theta (4~8Hz), Alpha (8~13Hz), Beta (13~30Hz) and Gamma (30~47Hz). It can be seen that by analyzing the EEG data, different emotion formation areas can be used for further joint analysis with facial data and eye movement data.

[0020] Regarding the collection of eye movement data, the collection of eye movement data is as follows: playing media resources with different emotional contents for the subjects to watch to record the eye movement trajectory of the subjects, and analyzing their attention and interest points under the media resources with different emotional contents; the subjects will increase their gaze time on the media resources that they are quite interested in, and the length of the gaze time is used to judge the subjects' attention and interest, and then the emotional content that the subjects are most interested in is calculated; Regarding facial data collection, the steps for facial data collection are as follows: Media resources with different emotional contents are played for the subjects to watch in order to capture the facial expressions of the subjects, so that the facial expressions are matched with the facial library data and then the facial data is output; facial data has a large amount of data for reference, therefore, by establishing a facial database, collecting the facial expressions of the subjects and matching them with the facial database, the emotions in the real-time state can be quickly obtained.

[0021] In the specific scheme, the media resources include video, audio or text; the emotional content includes fear content, anger content, sadness content and happiness content.

[0022] After the EEG data, facial data and eye movement data are accurately collected by the above technical means, they serve as the data basis for subsequent operations; in this embodiment, the weight calculation specifically includes the following stage steps: S1: Make a preliminary guess about the subject’s emotions based on facial data and obtain a preliminary judgment of the emotions; S2: Matching the initially judged emotion with the corresponding emotion in the EEG data to obtain a first judgment auxiliary value; S3: Matching the preliminary judged emotion with the corresponding emotion in the eye movement data to obtain a second judgement auxiliary value; S4: Obtain the highest value of the gradient standard value, perform weighted operation on the preliminary judgment emotion, the first judgment auxiliary value, and the second judgment auxiliary value to obtain an operation result, compare the operation structure with the standard value, and if the operation result is greater than the standard value, use the preliminary judgment value as the output emotion value and the trustworthiness of the emotion value; S5: If it is less than the operation structure, the secondary value of the gradient standard value is taken as the highest value, and the process loops into S4 and S5 until the sentiment value and the confidence of the sentiment value are output.

[0023] It is not difficult to understand that, because human emotions can often be quickly assisted by facial expressions, the collection and processing of facial data can have a more accurate data basis; the present application can quickly realize preliminary guesses through facial data, and further, the introduction of the first judgment auxiliary value and the second judgment auxiliary value can make the calculation of emotional values ​​more accurate, which can help users to achieve more efficient and accurate output of the emotional values ​​of the subjects, and based on the gradient standard value to correspond to the output trust, the user can get more objective feedback; it is worth understanding that the weighting coefficient of the first judgment value is greater than the weighting coefficient of the second judgment value, because the EEG data is more referenceable, and more accurate emotional values ​​can be obtained for the user to assist in judgment.

[0024] The present application also provides an emotion detection system based on the fusion of EEG signals, facial data and eye tracking technology, please refer to the attached Figure 1 , including an electronic headband, camera and processor; The electronic headband collects the subject’s EEG data; The camera collects facial data and eye movement data; The processor performs weighted calculations on the EEG data, facial data, and eye movement data to output an emotion index, and assists in determining the emotional state of the subject based on the emotion index; The electronic headband can accurately detect brain data, while the camera can capture eye movement data and facial data through algorithms; in the subsequent process, the processor calculates the collected EEG data, facial data and eye movement data to obtain the output emotion index, which is convenient for further inference of the emotional state of the subject. Since the EEG data, facial data and eye movement data are combined for comprehensive calculations, the user's emotion detection is more efficient and accurate; in terms of specific hardware installation, the processor can be installed on the electronic headband, and then the tablet, mobile phone or TV equipped with a camera can be called to collect facial data and eye movement data.

[0025] The present application also provides an information inference method for emotion detection based on the fusion of EEG signals, facial data and eye tracking technology, comprising the following steps: Collecting the EEG data, facial data and eye movement data of the subject, performing weight calculation on the EEG data, facial data and eye movement data to output an emotion index; Assist in judging the emotional state of the subject based on the emotional index; Collect emotional states and associate them with the user's historical preference database and content tag database to generate push information; Due to the comprehensive calculation of EEG data, facial data and eye movement data, the user's emotion detection is more efficient and accurate, and appropriate content can be pushed to users more appropriately to help users make emotional changes.

[0026] The advantages of the present invention are: By combining EEG data, facial data and eye movement data for comprehensive calculations, user emotion detection becomes more efficient and accurate.

[0027] The above disclosures are only several specific embodiments of the present invention, but the present invention is not limited thereto, and any changes that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. An emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology, characterized in that: The following steps are involved: Collecting EEG data, facial data and eye movement data of the subject, and performing weight calculation on the EEG data, the facial data and the eye movement data to output an emotion index; The emotional index is used to assist in judging the emotional state of the subject.

2. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to claim 1 is characterized in that: The steps of collecting the EEG data are as follows: Media resources with different emotional contents are played to the subjects to induce emotional data with different emotional characteristics, and the EEG data can be obtained by recording the EEG value of each emotional data.

3. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to claim 1 is characterized in that: The eye movement data is collected as follows: Media resources with different emotional contents are played to the subjects to record the subjects' eye movement trajectories, and their attention and interest points under the media resources with different emotional contents are analyzed.

4. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to claim 1 is characterized in that: The steps of collecting the facial data are as follows: Media resources with different emotional contents are played for the subjects to watch so as to capture the facial expressions of the subjects, so that the facial expressions are matched with the facial library data and then the facial data is output.

5. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to any one of claims 2 to 4, characterized in that: The media resources include video, audio or text.

6. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to any one of claims 2 to 4, characterized in that: The emotional content includes fear content, anger content, sadness content and happiness content.

7. The emotion detection method based on the fusion of EEG signals, facial data and eye tracking technology according to any one of claims 2 to 4, characterized in that: The weight calculation specifically includes: S1: Make a preliminary guess about the subject’s emotions based on facial data and obtain a preliminary judgment of the emotions; S2: Matching the initially judged emotion with the corresponding emotion in the EEG data to obtain a first judgment auxiliary value; S3: Matching the preliminary judged emotion with the corresponding emotion in the eye movement data to obtain a second judgement auxiliary value; S4: Obtain the highest value of the gradient standard value, perform weighted operation on the preliminary judgment emotion, the first judgment auxiliary value, and the second judgment auxiliary value to obtain an operation result, compare the operation structure with the standard value, and if the operation result is greater than the standard value, use the preliminary judgment value as the output emotion value and the trustworthiness of the emotion value; S5: If it is less than the operation structure, the secondary value of the gradient standard value is taken as the highest value, and the process loops into S4 and S5 until the sentiment value and the confidence of the sentiment value are output.

8. An emotion detection system based on the fusion of EEG signals, facial data and eye tracking technology, characterized in that: Includes electronic headband, camera and processor; The electronic headband collects EEG data of the subject; The camera collects facial data and eye movement data; The processor performs weighting operations on the EEG data, the facial data, and the eye movement data to output an emotion index, and assists in determining the emotional state of the subject based on the emotion index.

9. An information inference method for emotion detection based on the fusion of EEG signals, facial data and eye tracking technology, characterized in that: The following steps are involved: Collecting EEG data, facial data and eye movement data of the subject, and performing weight calculation on the EEG data, the facial data and the eye movement data to output an emotion index; Assist in judging the emotional state of the subject based on the emotional index; The emotional state is collected and associated with a user history preference database and a content tag database to generate push information.

Images