Video backtracking-based continuous emotion labeling method, system and electronic device

Abstract

The embodiment of the application provides a continuous emotion labeling method, system and electronic equipment based on video backtracking. The method comprises the following steps: using an evoking module to select evoking materials of multiple emotions from an emotion evoking material library and playing the evoking materials to a subject, collecting electroencephalogram signals of the subject when the subject watches the evoking materials through an electroencephalogram signal collecting module; determining linear smooth continuous electroencephalogram features from the electroencephalogram signals through a feature extraction module; guiding the subject to actively perform continuous scoring labeling of emotion evoking of a backtracking video of the evoking materials through a video backtracking module, and recording emotion evoking degrees corresponding to the evoking materials in time by an emotion labeling module; and determining continuous emotion labeling of the evoking materials based on the continuous electroencephalogram features and the emotion evoking degrees. The continuous emotion labeling method adopted by the embodiment of the application can make emotions correspond to videos in real time, and the continuous emotion labels can feed back continuous fluctuations of emotions of the subject, so that the determined emotion labels are more accurate, and the error of emotion labeling is reduced.

Description

Technical Field

[0001] This invention relates to the field of emotion brain-computer interface technology, and in particular to a continuous emotion annotation method, system and electronic device based on video backtracking. Background Technology

[0002] Emotions are an important part of daily human life, reflecting cognitive states and influencing psychology and behavior. With the rapid development of artificial intelligence and human-computer interaction, more human-like intelligent assistants are emerging to assist people in various fields. To further improve the user experience, intelligent assistants will be equipped with the ability to understand human emotions. To enable them to learn human emotions in different states, sufficient EEG signals labeled with emotions from humans in different states are needed for learning. EEG signals are important physiological signals of the human central nervous system and possess a strong ability to represent emotions.

[0003] Emotional induction is commonly used to obtain emotionally charged EEG signals. Emotional induction is a technique in social psychology experiments that uses emotionally evoked materials to activate corresponding emotions in subjects. Subjects are typically shown videos or images, and their EEG and other physiological signals are collected during this process. After watching a video clip, subjects complete a questionnaire to express their subjective feelings about the video, usually recording only a score. This score, or the inherent labeling attribute of the video itself, will be used as the label for the entire EEG and other physiological signal data. For example, if a subject is shown a video expressing happiness, and their subjective evaluation score after watching the video is high, then the EEG signal from watching the video expressing happiness will be labeled as a happy emotion.

[0004] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0005] For a video, using a single discrete emotion label to summarize emotional changes over 3-5 minutes can lead to labeling errors or significant score discrepancies because people don't maintain the same emotion throughout the viewing process. In other words, the given discrete label can only represent the participant's emotional state during certain time periods; the emotional state at other times cannot be described using this discrete label. Similarly, using one-time valence, arousal, and dominance scores can also result in significant errors in the scores of genuine emotional states. All these methods directly lead to a large amount of label noise in the model's input data, preventing the model from learning the emotion-related parts of the data. Specifically, videos may evoke more than one emotion; for example, videos that evoke "laughing in anger," "sadness after joy," or "crying with emotion." Suppose a participant watches a "Chinese New Year" video labeled with a happy emotion; ideally, the participant's EEG signal should be labeled with a happy emotion. During the playback of the "Chinese New Year" video, the participants experienced the joy and happiness of the holiday. However, at the end of the video, the emotions intensified as the participants, unfortunately unable to return home for the New Year and reunite with their families, felt a pang of sadness. At this point, the participants' EEG signals showed both happy and sad emotions, as did the sadness at the end. Labeling these EEG signals as happy emotions would be inaccurate, and models trained using inaccurately labeled EEG signals would struggle to accurately identify human emotions, resulting in lower learning capabilities. Summary of the Invention

[0006] To at least address the problems of emotion annotation in existing EEG signals, in a first aspect, embodiments of the present invention provide a continuous emotion annotation method based on video playback, comprising:

[0007] The evoked material module selects various emotion-evoking materials from the emotion-evoking material library and plays them to the subjects. The EEG signal collection module collects the EEG signals of the subjects while they are watching the evoked material.

[0008] The energy spectrum of the frequency domain signal is extracted from the EEG signal by the feature extraction module, and the linear smooth continuous EEG features are determined based on the energy spectrum.

[0009] After the subjects finished watching the evoked material, the video playback module guided the subjects to actively rate and label the emotional evoked by the playback video of the evoked material. The emotion labeling module recorded the degree of emotional evoked by the subjects corresponding to the time of the evoked material.

[0010] The continuous emotion labeling of the evoked material is determined based on the continuous EEG characteristics and the degree of emotion evoked.

[0011] Secondly, embodiments of the present invention provide a continuous emotion annotation system based on video backtracking, comprising:

[0012] The video-induced module is used to select various emotion-inducing materials from the emotion-inducing material library and play them to the subjects;

[0013] The electroencephalogram (EEG) signal collection module is used to collect the EEG signals of the subject while he / she is viewing the evoked material;

[0014] A preprocessing module is used to preprocess the electroencephalogram (EEG) signals;

[0015] The feature extraction module is used to extract the energy spectrum of the frequency domain signal from the EEG signal and determine the linearly smooth continuous EEG features based on the energy spectrum;

[0016] The video playback module is used to guide the subjects to actively and continuously score and label the playback videos of the evoking materials to induce emotions.

[0017] The emotion labeling module is used to record the degree of emotion evoked by the subject in relation to the time of the evoked material, and to determine the continuous emotion labeling of the evoked material based on the continuous EEG characteristics and the degree of emotion evoked.

[0018] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the video retrospective-based continuous sentiment annotation method of any embodiment of the present invention.

[0019] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the continuous sentiment annotation method based on video backtracking according to any embodiment of the present invention.

[0020] The beneficial effects of this invention are as follows: the continuous emotion labeling method enables real-time correspondence between emotions and videos. Continuous emotion labels can reflect the continuous fluctuations of the subject's emotions, making the determined emotion labels more accurate and reducing the error of emotion labeling. Therefore, in studies on the influence of EEG and other physiological signals on emotion induction, training the model using the emotion labeling method of this invention can effectively improve the accuracy of the model in emotion recognition. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a continuous emotion annotation method based on video backtracking provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the continuous emotion annotation results of a subject according to an embodiment of the continuous emotion annotation method based on video backtracking provided by the present invention;

[0024] Figure 3 This is a schematic diagram of the continuous emotion annotation results of multiple subjects in a continuous emotion annotation method based on video backtracking provided in an embodiment of the present invention;

[0025] Figure 4 This is a classification data map of different frequency bands of each subject before and after data screening based on thresholds, provided by an embodiment of the present invention for a continuous emotion annotation method based on video retrospection;

[0026] Figure 5 This is a schematic diagram of the structure of a continuous emotion annotation system based on video backtracking provided in an embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of an embodiment of an electronic device for continuous emotion annotation based on video backtracking, provided as an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] like Figure 1 The diagram shows a flowchart of a continuous sentiment annotation method based on video backtracking according to an embodiment of the present invention, which includes the following steps:

[0030] S11: Using the induction module, select various emotion-inducing materials from the emotion-inducing material library and play them to the subject. The EEG signal collection module collects the EEG signals of the subject while watching the inducing materials.

[0031] S12: Extract the energy spectrum of the frequency domain signal from the EEG signal through the feature extraction module, and determine the linearly smooth continuous EEG features based on the energy spectrum;

[0032] S13: After the subject watched the evoked material, the video playback module guided the subject to actively score and annotate the playback video of the evoked material to continuously evoke emotions. The emotion annotation module recorded the degree of emotion evoked by the subject corresponding to the time of the evoked material.

[0033] S14: Determine the continuous emotion labeling of the evoked material based on the continuous EEG characteristics and the degree of emotion evoked.

[0034] In this embodiment, considering that existing emotion labeling techniques typically involve subjects using a labeler to report their emotional state in real time during emotion induction, while this method can achieve real-time labeling of the subject's own emotional state, it essentially adds another labeling task to the emotion collection process, often affecting the induction of emotions. To avoid affecting the subjects during emotion induction, this method uses multiple subjects with an equal male-to-female ratio, good physical and mental health, and no mental illness (e.g., 8 subjects). The subjects are seated quietly in a private room with suitable temperature and humidity, and an emotion-inducing video is played on a computer monitor for them to watch.

[0035] For step S11, the induction module selects various emotion-inducing materials from the emotion-inducing material library and plays them to the subject. The emotion-inducing material library is constructed from video materials collected from the internet.

[0036] Specifically, emotionally evoking video materials can be collected from the internet. For example, video clips from social media forums can be crawled, with clip lengths set to 1-5 minutes. The video materials should primarily consist of movies, TV series, and documentaries. Movies, TV series, and documentaries are dynamic materials, which have the advantage of strong and rich emotions, resulting in a significant evoking effect. These collected video clips can evoke a variety of emotional responses in ordinary people.

[0037] The collected video footage was given to non-participating individuals from different age groups who were not involved in the emotional triggering process (meaning the participants did not participate in the scoring of the emotional triggering footage). To save on labor costs, the degree to which the video clips evoked personal emotions could also be scored based on the comments, bullet comments, and messages below the video clips. For example, the proportion of keywords such as "haha," "so infuriating," and "tears welling up" representing different emotional categories (happiness, sadness, disgust, fear, surprise, and anger) could be used to score the degree to which the video evoked personal emotions.

[0038] Videos were scored based on their ability to evoke individual emotions. A video library was created by selecting videos with high emotional evokedness and those with low emotional evokedness (representing neutral emotions). The evoked emotion module selected videos from this library to activate three types of emotions in the participants: negative, neutral, and positive.

[0039] The EEG signal collection module can collect the subject's EEG signals based on the ESI NeuroScan wet electrode EEG cap. The ESI NeuroScan wet electrode EEG cap can collect the subject's EEG signals at a frequency of 1000Hz when watching each video clip, thus obtaining the subject's EEG signals when watching the evoked material.

[0040] For step S12, the feature extraction module can use a fixed-length Hanning window to perform a fast Fourier transform on the obtained EEG signal to calculate the spectrum in the time and frequency domains.

[0041] As one implementation method, the EEG signal is subjected to a fast Fourier transform by a feature extraction module, which transforms the time-domain signal of the EEG signal into multiple frequency-domain signals, and the corresponding energy spectrum is determined based on the spectrum of the multiple frequency-domain signals.

[0042] The differential entropy feature of the energy spectrum is calculated, and the differential entropy feature is smoothed by linear dynamical system processing to obtain linearly smooth continuous EEG features.

[0043] The spectra of the multiple frequency domain signals include: delta spectrum, theta spectrum, alpha spectrum, beta spectrum, and gamma spectrum.

[0044] In this embodiment, the purpose of applying the Fourier transform is to convert a signal in the time domain into a signal in a different frequency domain. The transformation can be performed based on the following formula:

[0045]

[0046] Where ω represents frequency, t represents time, and e -iwt This represents a complex function.

[0047] The energy spectrum is determined by using the delta, theta, alpha, beta, and gamma spectra of multiple transformed frequency domain signals, where the k-th frequency ω k The energy spectrum is calculated as follows:

[0048] E(ω k )=X(ω k )X * (ωk )

[0049] Where X(●) represents the original signal, X * (●) represents the conjugate function of X(●).

[0050] The energy spectrum determined by the above formula is used to calculate the differential entropy feature of EEG characteristics. The formula for calculating the differential entropy feature h(x) using the energy spectrum is as follows:

[0051] h(x) = -∫ X f(x)log f(x)dx

[0052] Here, X is a random variable, and f(x) is the probability density function of X.

[0053] The differential entropy features identified above are subjected to feature smoothing to obtain linearly smooth continuous EEG features.

[0054] As one implementation, to further improve the accuracy of continuous EEG features, before extracting the energy spectrum of the frequency domain signal from the EEG signal through the feature extraction module, the method further includes:

[0055] The 50Hz AC power noise in the EEG signal is removed by filtering and the invalid signal in the EEG signal is removed by using a 1-75Hz bandpass filter to obtain the filtered EEG signal.

[0056] The filtered EEG signal is then marked and repaired for bad leads, and electrooculography (EOG) and electromyography (EMG) noise are removed to obtain a preprocessed EEG signal.

[0057] In this embodiment, the original EEG signal is downsampled to 200Hz, and a 1-50Hz bandpass filter is applied to remove AC power noise. A 1-75Hz bandpass filter is then used to filter out low-frequency and high-frequency invalid information, while also filtering out AC power noise. The filtered EEG signal is then marked and repaired for bad leads, and electrooculography (EOG) and electromyography (EMG) noise are removed based on the ICA (independent component analysis) algorithm. This further ensures the accuracy of EEG features in subsequent steps.

[0058] In step S13, after the subject finishes watching the evoked material and stops EEG data collection, the video playback module guides the subject to actively and continuously score and annotate the playback video of the evoked material to reflect emotional responses. For example, the EEG signal can be scored between 0 and 5 (not necessarily integers). During playback, the subject can control the volume and playback speed of the playback video using the volume and speed control knobs, supporting a maximum playback speed of 1.25-8 times. This improves the efficiency of the subject's continuous scoring and annotation. The subject can start and stop at any time. If the subject wants to modify an already annotated segment, they can return to the desired position using the video progress bar and modify the time segment they wish to edit.

[0059] As one implementation method, the subjects are guided to actively use the mouse wheel to continuously score and label the emotional evoked content in the retrospective video.

[0060] The emotion annotation module records the horizontal curve of the degree of emotion evoked by the subject using the mouse wheel, corresponding to the time of the evoking material.

[0061] In this embodiment, to further improve the convenience and efficiency of participants' scoring and annotation, considering that participants need to continuously score and annotate the emotional evoked state of the retrospective video of the evoking material, if participants continuously score, they need to perform a large number of scoring and annotation operations, which is inefficient, and adjacent EEG scores are prone to jumps. To solve the above problems, participants can be guided to use the mouse wheel for continuous scoring and annotation. Specifically, the retrospective video and continuous scoring and annotation are performed simultaneously. The mouse wheel starts at 0 points by default. During the retrospective video, participants slide the mouse wheel up and down to linearly adjust the emotional evoked score (0-5 points, which can be non-integer), obtaining a horizontal curve of the emotional evoked state corresponding to the time of the evoking material. After annotating a retrospective video segment, the annotated emotional evoked curve is automatically stored.

[0062] In step S14, the continuous EEG features are directly classified using a support vector machine corresponding to the emotions, yielding the prediction accuracy for each emotion category. Furthermore, the horizontal curves corresponding to the emotion evoked by the continuous EEG features are used to filter out highly evoked EEG data; for example, a threshold of 2.5 is used to filter out highly evoked EEG data. Data filtering is performed for the annotations of each subject. Obtaining accurate continuous emotion annotations for the evoked materials allows for the accurate identification of more emotion annotations and reduces noise in the emotion annotations. Figure 2 The image shows the continuous emotion annotation results of a subject in 15 videos, where R represents happy video segments, G represents neutral emotion segments, and B represents sad emotion segments.

[0063] To further determine the accuracy of continuous emotion labeling using this method and its practical effectiveness, for example, the emotion-evoking module selected 15 videos from an emotion-evoking material library and played them to the subjects. The EEG data from the last 6 videos, after filtering, were used as the test set to maintain fairness in the comparison. The training set used both pre- and post-filtered data for model training and comparison; specifically, the first 6 videos were used as the training set, and the middle 3 as the validation set. The model used a linear support vector machine. Figure 3 As shown, the same video evoked both uniformity and difference in emotions among different participants. For certain video scenes, all participants felt they were well-elicited, but for other scenes at different times, participants experienced different emotional responses.

[0064] like Figure 4 As shown, the average prediction accuracy of cross-validation with 8 subjects is presented, along with a comparison standard for Support Vector Machine (SVM) with linear kernels. It can be seen that, whether in a single frequency band or across the entire frequency band, the model trained using high-evoked EEG signals achieves better results, especially with a more significant improvement in the classification of low-frequency EEG signals.

[0065] As can be seen from this implementation method, the continuous emotion labeling method enables real-time correspondence between emotions and videos. Continuous emotion labels can reflect the continuous fluctuations of the subject's emotions, making the determined emotion labels more accurate and reducing the error of emotion labeling. Therefore, in studies on the influence of EEG and other physiological signals on emotion induction, training the model using the emotion labeling method of this method can effectively improve the accuracy of the model in emotion recognition.

[0066] like Figure 5 The diagram shown is a structural schematic of a continuous emotion annotation system based on video backtracking provided in an embodiment of the present invention. The system can execute the continuous emotion annotation method based on video backtracking described in any of the above embodiments and is configured in a terminal.

[0067] This embodiment provides a continuous emotion annotation system 10 based on video backtracking, which includes: a video induction module 11, an EEG signal collection module 12, a preprocessing module 13, a feature extraction module 14, a video backtracking module 15, and an emotion annotation module 16.

[0068] The video induction module 11 is used to select various emotion-inducing materials from the emotion-inducing material library and play them to the subject; the EEG signal collection module 12 is used to collect the EEG signals of the subject while watching the inducing materials; the preprocessing module 13 is used to preprocess the EEG signals; the feature extraction module 14 is used to extract the energy spectrum of the frequency domain signal from the EEG signals and determine the linear smooth continuous EEG features based on the energy spectrum; the video playback module 15 is used to guide the subject to actively continuously score and label the playback video of the inducing materials to induce emotions; the emotion labeling module 16 is used to record the degree of emotion induction corresponding to the time of the inducing materials labeled by the subject and determine the continuous emotion labeling of the inducing materials based on the continuous EEG features and the degree of emotion induction.

[0069] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the continuous sentiment annotation method based on video backtracking in any of the above method embodiments.

[0070] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0071] The evoked material module selects various emotion-evoking materials from the emotion-evoking material library and plays them to the subjects. The EEG signal collection module collects the EEG signals of the subjects while they are watching the evoked material.

[0072] The energy spectrum of the frequency domain signal is extracted from the EEG signal by the feature extraction module, and the linear smooth continuous EEG features are determined based on the energy spectrum.

[0073] After the subjects finished watching the evoked material, the video playback module guided the subjects to actively rate and label the emotional evoked by the playback video of the evoked material. The emotion labeling module recorded the degree of emotional evoked by the subjects corresponding to the time of the evoked material.

[0074] The continuous emotion labeling of the evoked material is determined based on the continuous EEG characteristics and the degree of emotion evoked.

[0075] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the continuous sentiment annotation method based on video backtracking in any of the above method embodiments is executed.

[0076] Figure 6 This is a schematic diagram of the hardware structure of an electronic device based on a continuous sentiment annotation method using video backtracking, as provided in another embodiment of this application. Figure 6 As shown, the device includes:

[0077] One or more processors 610 and memory 620, Figure 6 Taking a processor 610 as an example, the device for the continuous sentiment annotation method based on video retrospection may also include an input device 630 and an output device 640.

[0078] The processor 610, memory 620, input device 630, and output device 640 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0079] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the video-based continuous sentiment annotation method in the embodiments of this application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the video-based continuous sentiment annotation method in the above-described embodiments.

[0080] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0081] Input device 630 can receive input numerical or character information. Output device 640 may include display devices such as a display screen.

[0082] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, they execute the continuous sentiment annotation method based on video backtracking in any of the above method embodiments.

[0083] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0084] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0085] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the video backtracking-based continuous sentiment annotation method of any embodiment of this invention.

[0086] The electronic devices described in this application exist in various forms, including but not limited to:

[0087] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0088] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0089] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0090] (4) Other electronic devices with data processing functions.

[0091] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A continuous sentiment annotation method based on video backtracking, comprising: The evoked material module selects various emotion-evoking materials from the emotion-evoking material library and plays them to the subjects. The EEG signal collection module collects the EEG signals of the subjects while they are watching the evoked material. The energy spectrum of the frequency domain signal is extracted from the EEG signal by the feature extraction module, and the linear smooth continuous EEG features are determined based on the energy spectrum. After the subjects finished watching the evoked material, the video playback module guided the subjects to actively rate and label the emotional evoked by the playback video of the evoked material. The emotion labeling module recorded the degree of emotional evoked by the subjects corresponding to the time of the evoked material. The continuous emotion labeling of the evoked material is determined based on the continuous EEG characteristics and the degree of emotion evoked. The step of determining the continuous emotion labeling of the eliciting material based on the continuous EEG features and the degree of emotion elicitation includes: The continuous EEG features were classified into emotions to obtain the prediction accuracy of each emotion category. The scores of consecutive emotion categories are determined based on the horizontal curve of the prediction accuracy of each emotion category and the degree of emotion evoked by the time of the evoking material. The evoking material is labeled with consecutive emotion categories whose scores are higher than a preset threshold.

2. The method according to claim 1, wherein, Prior to extracting the energy spectrum of the frequency domain signal from the EEG signal via the feature extraction module, the method further includes: The 50Hz AC power noise in the EEG signal is removed by filtering and the invalid signal in the EEG signal is removed by using a 1-75Hz bandpass filter to obtain the filtered EEG signal. The filtered EEG signal is then marked and repaired for bad leads, and electrooculography (EOG) and electromyography (EMG) noise are removed to obtain a preprocessed EEG signal.

3. The method according to claim 1, wherein, The step of extracting the energy spectrum of the frequency domain signal from the EEG signal through the feature extraction module, and determining the linearly smooth continuous EEG features based on the energy spectrum, includes: The EEG signal is subjected to Fast Fourier Transform by the feature extraction module, which transforms the time-domain signal of the EEG signal into multiple frequency-domain signals. The corresponding energy spectrum is determined based on the spectrum of the multiple frequency-domain signals. The differential entropy feature of the energy spectrum is calculated, and the differential entropy feature is smoothed by linear dynamical system processing to obtain linearly smooth continuous EEG features.

4. The method according to claim 3, wherein, The spectra of the multiple frequency domain signals include: delta spectrum, theta spectrum, alpha spectrum, beta spectrum, and gamma spectrum.

5. The method according to claim 1, wherein, The method of guiding the subjects to actively rate and label the evoked emotions in the replay video of the evoking material using the video replay module includes: The subjects were guided to actively use the mouse wheel to continuously score and label the emotional evoked content in the retrospective video; The emotion annotation module records the horizontal curve of the degree of emotion evoked by the subject using the mouse wheel, corresponding to the time of the evoking material.

6. The method according to claim 1, wherein, The emotional trigger material library is constructed from video footage collected from the Internet.

7. A continuous sentiment annotation system based on video backtracking, comprising: The video-induced module is used to select various emotion-inducing materials from the emotion-inducing material library and play them to the subjects; The electroencephalogram (EEG) signal collection module is used to collect the EEG signals of the subject while he / she is viewing the evoked material; A preprocessing module is used to preprocess the electroencephalogram (EEG) signals; The feature extraction module is used to extract the energy spectrum of the frequency domain signal from the EEG signal and determine the linearly smooth continuous EEG features based on the energy spectrum; The video playback module is used to guide the subjects to actively and continuously score and label the playback videos of the evoking materials to induce emotions. The emotion labeling module is used to record the degree of emotion evoked by the subject in relation to the time of the evoking material, and to determine the continuous emotion labeling of the evoking material based on the continuous EEG characteristics and the degree of emotion evoked. The step of determining the continuous emotion labeling of the eliciting material based on the continuous EEG features and the degree of emotion elicitation includes: The continuous EEG features were classified into emotions to obtain the prediction accuracy of each emotion category. The scores of consecutive emotion categories are determined based on the horizontal curve of the prediction accuracy of each emotion category and the degree of emotion evoked by the time of the evoking material. The evoking material is labeled with consecutive emotion categories whose scores are higher than a preset threshold.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

9. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

Images