Emotional fluctuation event recording method and device and storage medium

Abstract

The invention provides an emotional fluctuation event recording method, an emotional fluctuation event recording device and a computer readable storage medium. The emotional fluctuation event recording method comprises the following steps: acquiring an electroencephalogram signal of a user to be recorded; according to the electroencephalogram signal, determining a probability value used for representing that the emotion of the to-be-recorded user originates from a virtual event; in response to the fact that the probability value is smaller than a preset first threshold value, it is judged that the emotion fluctuation of the user to be recorded is caused by a real event, and event content of the real event is recorded. The emotion fluctuation caused by the virtual event and the real event can be distinguished by judging the emotion source of the user, and the emotion intensity of the user is dynamically calculated through the multi-modal signal and is used for recording the event content of the real event causing the emotion fluctuation of the user, so that the authenticity of the recorded content in the emotion recognition process is ensured, and the user experience is improved. In this way, erroneous records related to emotional fluctuations caused by virtual events are eliminated.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular, to a method for recording emotion fluctuation events, a device for recording emotion fluctuation events, and a computer-readable storage medium. Background Art

[0002] With the rapid development of intelligent technologies, people's demands for emotional expression and life recording have become increasingly refined and personalized. As a window directly reflecting brain activities, electroencephalogram (EEG) signals are often used to identify emotional states for recording the event content that causes emotional fluctuations of users. However, existing shooting devices for recording emotion fluctuation events focus on classifying the recognized emotions themselves, ignoring the sources of emotions and changes in emotion intensity. As a result, virtual events such as watching videos, chatting, and recollecting that do not need to be recorded may be misrecorded, thereby reducing the efficiency in the process of emotion recognition and analysis.

[0003] In order to overcome the above-mentioned defects existing in the prior art, there is an urgent need in this field for an improved method for recording emotion fluctuation events, which is used to record the event content of real events that cause emotional fluctuations of users, so as to ensure the authenticity of the recorded content in the process of emotion recognition and exclude false records of emotional fluctuations caused by virtual events. Summary of the Invention

[0004] The following provides a brief overview of one or more aspects to provide a basic understanding of these aspects. This overview is not an exhaustive survey of all contemplated aspects, and neither is it intended to identify key or decisive elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description to follow.

[0005] In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a method for recording emotion fluctuation events, a device for recording emotion fluctuation events, and a computer-readable storage medium. It can distinguish emotional fluctuations caused by virtual events and real events by judging the source of users' emotions, and dynamically calculate the intensity of users' emotions through multimodal signals, so as to record the event content of real events that cause emotional fluctuations of users, thereby ensuring the authenticity of the recorded content in the process of emotion recognition and excluding false records of emotional fluctuations caused by virtual events.

[0006] Specifically, the method for recording the above-mentioned emotional fluctuation events provided by the first aspect of the present invention includes the following steps: obtaining the electroencephalogram (EEG) signal of the user to be recorded; determining, based on the EEG signal, a probability value for characterizing that the emotion of the user to be recorded stems from a virtual event; and in response to the probability value being less than a preset first threshold, determining that the emotional fluctuation of the user to be recorded is caused by a real event, and recording the event content of the real event.

[0007] Further, in some embodiments of the present invention, the EEG signal includes alpha waves, beta waves, gamma waves, delta waves, and theta waves. The step of determining, based on the EEG signal, a probability value for characterizing that the emotion of the user to be recorded stems from a virtual event includes: using a filter to remove the noise signals in each of the EEG signals. The noise signals at least include power frequency interference generated when the power system operates and electromyographic noise; and performing frequency domain feature analysis on each of the EEG signals to obtain the power spectral density of each of the EEG signals, so as to extract the eigenvalue of each of the EEG signals; and determining the probability value based on the eigenvalues of the alpha waves, the beta waves, the gamma waves, the delta waves, and the theta waves.

[0008] Further, in some embodiments of the present invention, the step of determining, based on the EEG signal, a probability value for characterizing that the emotion of the user to be recorded stems from a virtual event further includes: obtaining the environmental signal, facial expression signal, and physiological signal of the user to be recorded; preprocessing the environmental signal, facial expression signal, and physiological signal to obtain the eigenvalues of the environmental signal, facial expression signal, and physiological signal; and normalizing the eigenvalues of the environmental signal, facial expression signal, and physiological signal to map each of the eigenvalues to the same range; and determining the probability value based on the eigenvalues of each of the EEG signals, the environmental signal, the facial expression signal, and the physiological signal.

[0009] Further, in some embodiments of the present invention, the step of in response to the probability value being less than a preset first threshold, determining that the emotional fluctuation of the user to be recorded is caused by a real event, and recording the event content of the real event includes: determining an emotional intensity value for characterizing the emotional intensity of the user to be recorded based on the EEG signal, the environmental signal, the facial expression signal, and the physiological signal; and in response to the emotional intensity value being greater than a preset second threshold and its fluctuation duration being greater than a preset third threshold, triggering a first instruction to photograph the user to be recorded to record the event content of the real event.

[0010] Further, in some embodiments of the present invention, the step of determining the emotion intensity value used to characterize the emotion intensity of the user to be recorded based on the EEG signal, the environmental signal, the facial expression signal and the physiological signal includes: respectively obtaining initial first weight values ​​of the EEG signal, the environmental signal, the facial expression signal and the physiological signal; adjusting each of the first weight values ​​according to the variation amplitude of the characteristic value of the EEG signal, the environmental signal, the facial expression signal and / or the physiological signal, so as to respectively determine the second weight values ​​of the EEG signal, the environmental signal, the facial expression signal and the physiological signal:

[0011]

[0012] in, is the first weight value of the i-th signal. i is the second weight value of the i-th signal. i is the variation range of the characteristic value of the i-th signal. α is the weight adjustment factor; and the emotion intensity value is determined according to each of the second weight values ​​and the characteristic value of each of the signals:

[0013]

[0014] Among them, F i is the eigenvalue of the ith signal.

[0015] Further, in some embodiments of the present invention, the step of adjusting the first weight value of the EEG signal according to the change of the EEG signal to determine the second weight value of the EEG signal includes: determining the weight adjustment value of the EEG signal according to the weight adjustment factor and the characteristic value of the EEG signal:

[0016]

[0017] Where, ΔW EEG is the weight adjustment value of the EEG signal. α is the weight adjustment factor. E EEG is the characteristic value of the EEG signal; and according to the first weight value and the weight adjustment value, determining a second weight value of the EEG signal:

[0018]

[0019] in, is the first weight value of the EEG signal. EEG is the second weight value of the EEG signal.

[0020] Further, in some embodiments of the present invention, the method for recording the emotional fluctuation event further includes the following steps: in response to the probability value being greater than the first threshold, adjusting the emotional intensity value according to the probability value:

[0021]

[0022] wherein, is the adjusted emotional intensity value. I emotion is the emotional intensity value before adjustment. P virtual is the probability value.

[0023] Further, in some embodiments of the present invention, after triggering the first instruction to photograph the user to be recorded to record the event content of the real event, the following steps are further included: automatically generating an emotional fluctuation analysis report of the user to be recorded according to the video data and / or audio data during the emotional fluctuation of the user to be recorded. The emotional fluctuation analysis report at least includes the fluctuation duration, peak intensity, and occurrence time of the emotional fluctuation.

[0024] Further, in some embodiments of the present invention, after triggering the first instruction to photograph the user to be recorded to record the event content of the real event, the following steps are further included: in response to the probability value being greater than the first threshold, or the emotional intensity value being less than a preset fourth threshold and its fluctuation duration being less than the third threshold, or the change amplitudes of the signal feature values of each of the electroencephalogram signals, environmental signals, facial expression signals, and physiological signals being lower than a preset fifth threshold, triggering a second instruction to stop photographing the user to be recorded.

[0025] In addition, the above-mentioned recording device for emotional fluctuation events provided by the second aspect of the present invention includes a signal acquisition module, a data processing module, and a photographing module. The signal acquisition module is used to acquire various signals of the user to be recorded. The signal acquisition module at least includes an electroencephalogram sensor, an environmental sensor, a camera, and a smart wearable device. The signals at least include electroencephalogram signals, environmental signals, facial expression signals, and physiological signals. The data processing module is used to determine the probability value for characterizing that the emotion of the user to be recorded originates from a virtual event according to the various signals. The photographing module is used to record the event content of the real event that causes the emotional fluctuation of the user to be recorded.

[0026] Further, in some embodiments of the present invention, the recording device for emotional fluctuation events further includes a memory and a controller. A computer instruction is stored on the memory. The controller is connected to the memory, the signal acquisition module, the data processing module, and the photographing module, and is used to execute the computer instruction stored on the memory to implement the method for recording emotional fluctuation events provided by the first aspect of the present invention.

[0027] In addition, the above computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions. When the computer instructions are executed by a processor, a method for recording mood swing events provided as in the first aspect of the present invention is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] After reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings, the above features and advantages of the present invention can be better understood. In the drawings, the components are not necessarily drawn to scale, and components having similar relevant characteristics or features may have the same or similar reference numerals.

[0029] Figure 1 The structural schematic diagram of a device for recording mood swing events provided according to some embodiments of the present invention is shown.

[0030] Figure 2 The flowchart of a method for recording mood swing events provided according to some embodiments of the present invention is shown.

[0031] Figure 3 The flowchart of a method for recording mood swing events provided according to some embodiments of the present invention is shown.

[0032] Reference Numerals:

[0033] 11 Signal Acquisition Module

[0034] 111 Electroencephalogram Sensor

[0035] 112 Environment Sensor

[0036] 113 Camera

[0037] 114 Smart Wearable Device

[0038] 12 Data Processing Module

[0039] 13 Power Management Module

[0040] 14 Ring Buffer Storage Module

[0041] 15 Persistent Storage Module

[0042] 16 User Interface DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiments, this does not mean that the features of this invention are limited to this implementation manner. On the contrary, the purpose of introducing the invention in conjunction with the implementation manner is to cover other alternatives or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, many specific details will be included in the following description. The present invention can also be implemented without using these details. In addition, in order to avoid confusing or obscuring the key points of the present invention, some specific details will be omitted in the description.

[0044] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "mounted", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0045] In addition, the "upper", "lower", "left", "right", "top", "bottom", "horizontal", and "vertical" used in the following description should be understood as the orientations shown in this paragraph and the related drawings. This relative term is only for the convenience of description, and it does not mean that the device described needs to be manufactured or operated in a specific orientation. Therefore, it should not be understood as a limitation to the present invention.

[0046] It can be understood that although the terms "first", "second", "third", etc. can be used here to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first component, region, layer, and / or part discussed below can be called the second component, region, layer, and / or part without departing from some embodiments of the present invention.

[0047] As described above, with the rapid development of intelligent technologies, people's demands for emotional expression and life recording have become increasingly refined and personalized. Electroencephalogram (EEG) signals, as a window directly reflecting brain activities, are often used to identify emotional states to record the content of events that trigger users' emotional fluctuations. However, existing shooting devices for recording emotional fluctuation events focus on classifying the recognized emotions themselves, ignoring the sources of emotions and changes in emotional intensity. As a result, virtual events that do not need to be recorded, such as watching videos, chatting, and recollecting, may be misrecorded, thus reducing the efficiency in the process of emotion recognition and analysis.

[0048] To overcome the above-mentioned defects existing in the prior art, the present invention provides a method for recording emotional fluctuation events, a device for recording emotional fluctuation events, and a computer-readable storage medium. It can distinguish emotional fluctuations caused by virtual events and real events by judging the source of users' emotions, and dynamically calculate the users' emotional intensity through multimodal signals, so as to record the event content of real events that trigger users' emotional fluctuations, thereby ensuring the authenticity of the recorded content in the process of emotion recognition and excluding false records regarding emotional fluctuations caused by virtual events.

[0049] In some non-limiting embodiments, the method for recording emotional fluctuation events provided in the first aspect of the present invention can be implemented based on the device for recording emotional fluctuation events provided in the second aspect of the present invention.

[0050] Specifically, please refer to Figure 1 。 Figure 1 FIG. shows a schematic structural diagram of a device for recording emotional fluctuation events provided in some embodiments of the present invention.

[0051] In Figure 1 the illustrated embodiment, the device for recording emotional fluctuation events provided in the second aspect of the present invention includes a signal acquisition module 11, a data processing module 12, and a shooting module (not shown). Herein, the signal acquisition module 11 is used to acquire various signals of the user to be recorded. The signal acquisition module 11 at least includes an EEG sensor 111, an environmental sensor 112, a camera 113, and an intelligent wearable device 114, and the signals at least include EEG signals, environmental signals, facial expression signals, and physiological signals. The data processing module 12 is used to determine a probability value for characterizing that the emotion of the user to be recorded stems from a virtual event according to the various signals. The data processing module 12 includes a processor such as ESP32 or STM32, an analog-to-digital converter (ADC), and a data fusion processing unit. The shooting module is used to record the event content of real events that trigger the emotional fluctuations of the user to be recorded.

[0052] Specifically, the EEG sensor 111 can collect EEG signals of the user to be recorded from the forehead or other specific areas of the user to be recorded. The environmental sensor 112 is used to collect environmental signals such as sound and light in the surrounding environment of the user to be recorded. The camera 113 is used to capture facial expression signals such as facial expressions. The smart wearable device 114 is used to collect physiological signals such as the heart rate and skin conductivity of the user to be recorded.

[0053] In addition, in Figure 1 the embodiment shown, the recording device for emotional fluctuation events provided in the second aspect of the present invention further optionally includes a power management module 13, a circular buffer storage module 14, a persistent storage module 15, and a user interface 16. Here, the power management module 13 includes a battery management integrated circuit and a lithium battery. The circular buffer storage module 14 includes a high-speed storage DDR (Double Data Rate) memory, and the persistent storage module 15 includes an eMMC or an SD card. The circular buffer storage module 14 cooperates with the persistent storage module 15 to record the event content data obtained by the shooting module. The user interface 16 obtains an emotional fluctuation analysis report generated based on the event content through wireless communication and displays the analysis report to the user.

[0054] In addition, in some non-limiting embodiments, the recording device for emotional fluctuation events provided in the second aspect of the present invention further includes a memory and a controller. Here, the memory includes, but is not limited to, the computer-readable storage medium provided in the aforementioned third aspect, on which computer instructions are stored. The controller is connected to the aforementioned memory, the signal acquisition module 11, the data processing module 12, and the shooting module, and is configured to execute the computer instructions stored on the memory to implement the recording method for emotional fluctuation events provided in the first aspect of the present invention.

[0055] The working principle of the above-mentioned recording device for emotional fluctuation events will be described below in conjunction with some embodiments of the recording method for emotional fluctuation events. Those skilled in the art can understand that these embodiments of the recording method are only some non-limiting implementation manners provided by the present invention, aiming to clearly show the main concept of the present invention and provide some specific solutions convenient for the public to implement, rather than limiting all functions or all working modes of the recording device. Similarly, the recording device for emotional fluctuation events is also a non-limiting implementation manner provided by the present invention, and does not limit the execution subject and execution order of each step in these recording methods for emotional fluctuation events.

[0056] Specifically, please refer to Figure 2 and Figure 3 . Figure 2 shows a schematic flowchart of a recording method for emotional fluctuation events provided in some embodiments of the present invention.Figure 3 A flowchart showing a method for recording mood swing events provided according to some embodiments of the present invention is presented.

[0057] As Figure 2 and Figure 3 shown, the above-mentioned recording device provided by the second aspect of the present invention can first obtain the electroencephalogram (EEG) signal of the user to be recorded via the EEG sensor 111 in the information acquisition module 11, and record the collected data via the circular buffer in the circular buffer storage module 14.

[0058] Specifically, the above-mentioned EEG signal includes alpha waves, beta waves, gamma waves, delta waves and theta waves. The frequency band range of delta waves is 0.5 Hz to 4 Hz, which is related to deep sleep and unconscious states. The frequency band of theta waves is 4 Hz to 8 Hz, which is related to memory, imagination and relaxation states. The frequency band range of alpha waves is 8 Hz to 12 Hz, which reflects the background brain wave activity during relaxation or concentration. The frequency band range of beta waves is 12 Hz to 30 Hz, which is related to attention, anxiety and cognitive load. The frequency band range of gamma waves is 30 Hz to 100 Hz, which is related to high-level cognitive processing, perception and emotion integration.

[0059] After that, the recording device can determine a probability value for characterizing that the mood of the user to be recorded stems from a virtual event based on the EEG signal.

[0060] Further, in some embodiments, the recording device can use a filter to remove the noise signals in each EEG signal. Here, the noise signal at least includes power frequency interference (e.g., 50 Hz power frequency interference) generated when the power system operates and electromyographic noise.

[0061] After that, the recording device can perform frequency domain feature analysis on each EEG signal, convert each EEG signal from the time domain to the frequency domain to obtain the power spectral density of each EEG signal, thereby extracting the characteristic values of each EEG signal, and further using them to represent the physiological and psychological information carried by each brain wave of different frequencies in the EEG signal.

[0062] After that again, the recording device can determine the probability value according to the characteristic values of alpha waves, beta waves, gamma waves, delta waves and theta waves.

[0063] Specifically, mood swings caused by virtual events such as making a phone call, watching an entertainment program, and recalling chatting with others are usually accompanied by relatively high epsilon wave activity due to imagination and recollection, and gradually decreasing gamma wave activity due to insufficient real sensory stimuli. On the contrary, mood swings caused by real events are usually accompanied by relatively high gamma wave activity and synchronous alpha wave or beta wave activity, reflecting actual sensory input and emotional responses.

[0064] Thus, the above recording device provided by the second aspect of the present invention can distinguish whether the source of emotional fluctuations is a real event or a virtual event based only on the characteristic values of various brain waves in the electroencephalogram signal.

[0065] In addition, as Figure 3 shown, the recording device can also acquire the environmental signal, facial expression signal, and physiological signal of the user to be recorded through the environmental sensor 112, camera 113, and smart wearable device 114 in the information acquisition module 11, respectively.

[0066] After that, the recording device can preprocess the environmental signal, facial expression signal, and physiological signal to obtain the characteristic values of the environmental signal, facial expression signal, and physiological signal.

[0067] Specifically, the recording device can use a microphone to collect the sound signal around the user to be recorded and perform spectrum analysis (FFT, Fast Fourier Transform) on it to extract features such as the sound intensity and frequency analysis of the sound signal.

[0068] In addition, the recording device can identify the facial feature points of the user to be recorded through computer vision image processing libraries such as OpenCV or Dlib to calculate facial expression features such as the smile index and the amplitude of eyebrow raising of the face.

[0069] In addition, the recording device can use smart watches, smart bracelets, etc. to collect information such as heart rate and skin conductivity for assisting in the analysis of emotional fluctuations.

[0070] After that, the recording device can normalize the characteristic values of the above environmental signal, facial expression signal, and physiological signal to map each characteristic value to the same range (for example: [0,1]).

[0071] After that, the recording device can determine the probability value according to the characteristic values of each electroencephalogram signal, environmental signal, facial expression signal, and physiological signal.

[0072] Specifically, please refer to Table 1. Table 1 shows the characteristic difference table of various signals generated by emotional fluctuations caused by virtual events and real events provided by some embodiments of the present invention.

[0073] Table 1 Characteristic Difference Table of Various Signals Generated by Emotional Fluctuations Caused by Virtual Events and Real Events

[0074]

[0075] Thus, the above recording device provided by the second aspect of the present invention can further combine the characteristic values of other multi-source signals to distinguish whether the source of emotional fluctuations is a real event or a virtual event, so as to improve the accuracy of distinguishing the emotional source.

[0076] Subsequently, in response to the probability value being less than a preset first threshold (e.g., 0.2), the recording device may determine that the emotional fluctuation of the user to be recorded is caused by a real event and record the event content of the real event.

[0077] Furthermore, as Figure 3 shown, the recording device may determine an emotional intensity value for characterizing the emotional intensity of the user to be recorded according to the electroencephalogram signal, the environmental signal, the facial expression signal, and the physiological signal.

[0078] Specifically, the recording device may respectively obtain the initial first weight values of the electroencephalogram signal, the environmental signal, the facial expression signal, and the physiological signal. Here, the recording device may set a relatively high first weight value for the electroencephalogram signal (e.g., 0.6), and relatively low values for other signals (e.g., 0.2 for the environmental signal, 0.1 for the facial expression signal, and 0.1 for the physiological signal).

[0079] Furthermore, in the process of determining the initial first weight values of the signals, the recording device may collect the signal data and the label data of the emotional changes in various scenarios, and the label includes the actual contribution values of various signals to the emotional judgment. After that, the recording device may use statistical methods to analyze the strong correlations of different signals. For example, in the movie-watching scenario, the correlation between the electroencephalogram signal and the emotional fluctuation is higher, while during skydiving, the sudden change contribution of the physiological signal is greater.

[0080] In addition, in some preferred embodiments, the recording device may train machine learning models such as multi-modal fusion networks and regression analysis for mapping the contribution degrees of different signals to the emotional judgment.

[0081] In addition, in some alternative embodiments, in the case of lack of sufficient training data, the recording device may first set the initial first weight values based on the prior knowledge of the importance of the signals. For example, EEG usually plays a dominant role in emotional judgment. After that, in a dynamic scenario, the weights of the physiological signal and the environmental signal should be increased. Conversely, in a static scenario, the weights of the electroencephalogram signal and the facial expression signal should be increased.

[0082] After that, in the Figure 3 shown embodiment, the recording device may adjust each first weight value according to the change amplitude of the electroencephalogram signal, the environmental signal, the facial expression signal, and / or the physiological signal feature value, so as to respectively determine the second weight values of the electroencephalogram signal, the environmental signal, the facial expression signal, and the physiological signal:

[0083]

[0084] Wherein, is the first weight value of the i-th type of signal, W i is the second weight value of the i-th type of signal, ΔFi is the change amplitude of the i-th signal eigenvalue, and α is the weight adjustment factor.

[0085] For example, when the fluctuations of the internal emotional signals of the user to be recorded are intense, the recording device can increase the weight of the electroencephalogram signals. When the environmental interference signals increase, such as when the noise or light changes frequently, the weight of the environmental signals can be increased. When the facial expressions change significantly, such as when the eyebrows are raised greatly, the weight of the facial expression signals can be increased.

[0086] Further, in the process of determining the second weight value of the electroencephalogram signals, the recording device can determine the weight adjustment value of the electroencephalogram signals according to the weight adjustment factor and the eigenvalue of the electroencephalogram signals:

[0087]

[0088] where ΔW EEG is the weight adjustment value of the electroencephalogram signals, α is the weight adjustment factor, and E EEG is the eigenvalue of the electroencephalogram signals.

[0089] After that, the determining device can determine the second weight value of the electroencephalogram signals according to the first weight value and the weight adjustment value:

[0090]

[0091] where is the first weight value of the electroencephalogram signals, and W EEG is the second weight value of the electroencephalogram signals.

[0092] After that, the recording device can input the second weight values and the eigenvalues of each signal into a pre-trained deep learning model (for example: LSTM model or Transformer model) to determine the emotion intensity value:

[0093]

[0094] where F i is the eigenvalue of the i-th signal.

[0095] After that, in response to the emotion intensity value being greater than a preset second threshold and its fluctuation duration being greater than a preset third threshold (for example: 3s, 30s, 1min), the recording device can trigger the first instruction to photograph the user to be recorded to record the event content of the real event.

[0096] Here, the setting range of the second threshold for emotion intensity triggering is [0.6, 0.9]. The larger the second threshold, the greater the user emotion intensity required for the recording device to trigger the photographing module for event recording, and the more intense the user emotion fluctuations recorded.

[0097] In addition, in some preferred embodiments, in response to the probability value being greater than the first threshold, the recording device can also adjust the emotion intensity value according to the probability value:

[0098]

[0099] Wherein, is the adjusted emotion intensity value, I emotion is the emotion intensity value before adjustment, P virtual is the virtual emotion probability.

[0100] In this way, the above-mentioned recording device provided by the second aspect of the present invention can, through the dynamic adjustment of the emotion intensity, when the emotion fluctuation is due to a relatively large probability value of a virtual event, make the corresponding emotion intensity value decrease, thereby avoiding mis-triggering the shooting module to record the event content of the virtual event.

[0101] Furthermore, as Figure 3 shown, in response to meeting the stop shooting condition, the recording device can trigger a second instruction to close the shooting for recording the user, and store the recording content during the shooting in the persistent storage module 15.

[0102] Even further, in some embodiments, in response to the probability value being greater than the first threshold, the recording device can trigger a second instruction to close the shooting for recording the user. Here, after the recording device determines that the emotion fluctuation is due to a virtual event, it does not start or close the shooting module, and enters the low-power mode, only retaining the emotion detection function.

[0103] Alternatively, in some embodiments, in response to the emotion intensity value being less than a preset fourth threshold and the duration of its fluctuation being less than a third threshold, or the change amplitude of the signal characteristic values of each of the electroencephalogram signal, environmental signal, facial expression signal, and physiological signal being lower than a preset fifth threshold, a second instruction to close the shooting for recording the user is triggered. Here, after the emotion fluctuation intensity of the recording device is not large or the emotion tends to be stable, it does not start or close the shooting module, and automatically saves the recording content.

[0104] After that, as Figure 3 shown, the recording device can automatically generate an emotion fluctuation analysis report of the user to be recorded based on the video data and / or audio data during the emotion fluctuation of the user to be recorded, and display it to the user via the user interface 16. Here, the emotion fluctuation analysis report at least includes the fluctuation duration, peak intensity, and occurrence time of the emotion fluctuation.

[0105] Thus, the above-mentioned recording device provided by the second aspect of the present invention can utilize a passive brain-computer interface (BCI) to automatically record complete events that induce strong emotional fluctuations in the user's life without the user consciously triggering the recording. By collecting and analyzing the user's unconscious brain electrical signals, the burden of use is reduced.

[0106] For example, in the field of psychological therapy assistance, the recording device can capture the triggering events of various emotional fluctuations in the patient's life, providing a basis for psychological assessment.

[0107] Again, for example, in the field of criminal investigation, due to the particularity of the crime scene, it may be difficult for eyewitnesses to record key evidence of the crime scene in a timely and proactive manner using traditional devices (such as mobile phone cameras). Eyewitnesses usually experience significant emotional fluctuations such as surprise, fear, and tension when witnessing sudden and unusual criminal acts. Therefore, the above-mentioned recording device provided by the second aspect of the present invention can trigger full-automatic operation and shooting based on emotional fluctuations in emergency situations without the need for the user to actively operate, ensuring the secrecy, timeliness, and integrity of evidence collection, thus providing support for solving cases.

[0108] Again, for example, in the field of security monitoring, due to the particularity of extreme sports such as rock climbing, skydiving, skiing, and diving, participants are unable to operate the camera device in real time for recording. Therefore, the above-mentioned recording device provided by the second aspect of the present invention can trigger shooting based on strong emotional fluctuations such as fear, excitement, and tension to automatically record precious images. In addition, if the participant faces sudden mistakes or dangerous situations, and the emotional fluctuations show signals such as fear and tension maintaining a high value for a long time indicating a dangerous situation, the recording device can send the captured images to the ground safety team through a connected communication device, facilitating the rescue team to understand the dangerous situation and provide appropriate rescue in a timely manner.

[0109] In order to verify the accuracy of the above-mentioned method, device, and storage medium for recording emotional fluctuation events provided by the present invention in analyzing the source of emotions, technicians can use virtual events and real events that cause emotional fluctuations in users for testing respectively.

[0110] For example, a user wears the recording device for emotional fluctuation events provided by the second aspect of the present invention and is watching a movie. The recording device can collect the user's brain electrical signals, environmental signals, facial expression signals, and physiological signals for analyzing the source of the user's emotional fluctuations.

[0111] Here, in the user's electroencephalogram (EEG) signals, alpha waves, beta waves, and theta waves are dominant, and the activity of gamma waves is relatively low. In the user's environmental signals, the sound signal is a stable background sound such as movie dialogue and sound effects, and its frequency characteristics are continuous and consistent. The light signal is a regular light change emitted by the screen, without sudden flashes or obvious interference. In the user's facial expression signals, the overall expression amplitude is small and the change is slow. The user's physiological signals indicate that the user is in a static state and the eyes are continuously fixed on the screen direction.

[0112] In this way, the recording device can combine the characteristic values of the above signals to determine that the probability value of the emotional fluctuation originating from the virtual event is 0.85, which is much greater than the preset first threshold (for example: 0.2). Therefore, the recording device can determine that the emotional fluctuation is caused by the virtual event.

[0113] After that, the recording device can set the first weight value of the EEG signal to 0.6, the first weight value of the environmental signal to 0.3, and the sum of the first weight values of the facial expression signal and the physiological signal to 0.1. Then, the recording device adjusts the above signals. After adjustment, the second weight value of the EEG signal is 0.4, the second weight value of the environmental signal is 0.5, and the sum of the second weight values of the facial expression signal and the physiological signal is 0.1.

[0114] Then, the recording device can calculate the emotional intensity value:

[0115] I emotion = 0.4×0.8 + 0.5×0.2 + 0.1×0.9 = 0.51

[0116] In response to the probability value of the emotional fluctuation originating from the virtual event being greater than the first threshold, adjust the emotional intensity value:

[0117]

[0118] In this way, in response to the probability value of the emotional fluctuation originating from the virtual event being greater than the first threshold and the emotional intensity value being less than the second threshold (for example: 0.7), the shooting module is not triggered to turn on, and the event content of the virtual event that causes the user's emotional fluctuation is not recorded, thereby avoiding unnecessary shooting records and ensuring storage efficiency and privacy protection. This process realizes the efficient combination of emotional source classification and shooting logic, highlighting the intelligence and applicability of the recording device for the above emotional fluctuation events provided by the second aspect of the present invention.

[0119] For another example, when the user wears the recording device for emotional fluctuation events provided by the second aspect of the present invention and participates in a skydiving sport. The recording device can collect the user's EEG signals, environmental signals, facial expression signals, and physiological signals for analyzing the source of the user's emotional fluctuation.

[0120] Here, in the user's EEG signals, gamma waves and beta waves are dominant, while the activities of alpha waves and theta waves are significantly reduced, indicating that the user is in a highly tense and excited state. In the user's environmental signals, the noise of the wind in the sound signal increases sharply, and its spectrum is irregular, accompanied by a brief silence at the moment of jumping out of the hatch. The light signal changes from a dim area inside the cabin to a strong light area in the air, with a significant change in light intensity. In the user's facial expression signals, the facial muscles are vigorously active and the eyes are wide open. In the user's physiological signals, the IMU sensor detects a significant acceleration change at the moment of jumping down, and the user's attitude angle is quickly adjusted.

[0121] In this way, the recording device can combine the characteristic values of the above signals to determine that the probability value of this emotional fluctuation originating from a virtual event is 0.15, which is less than a pre-set first threshold (for example: 0.2). Therefore, the recording device can determine that this emotional fluctuation is caused by a real event.

[0122] After that, the recording device can set the first weight value of the EEG signal to 0.4, the first weight value of the environmental signal to 0.3, and the sum of the first weight values of the facial expression signal and the physiological signal to 0.3. Then, the recording device adjusts the above signals. After adjustment, the second weight value of the EEG signal is 0.4, the second weight value of the environmental signal is 0.3, and the sum of the second weight values of the facial expression signal and the physiological signal is 0.3.

[0123] After that, the recording device can calculate the emotional intensity value:

[0124] I emotion = 0.4×0.9 + 0.3×0.8 + 0.3×0.75 = 0.825

[0125] In this way, in response to the probability value of the emotional fluctuation originating from a virtual event being less than the first threshold and the emotional intensity value being greater than the second threshold (for example: 0.7), the shooting module is triggered to start, and the event content of the real event that causes the user's emotional fluctuation is recorded, recording the key moment of the user's skydiving. This process highlights the intelligent triggering ability of the above-mentioned recording device for emotional fluctuation events provided by the second aspect of the present invention in a real high-dynamic scenario, provides precious images for the user, and ensures the efficient operation of the device.

[0126] In summary, the above-mentioned recording method for emotional fluctuation events, the recording device for emotional fluctuation events, and the storage medium provided by the present invention can all distinguish emotional fluctuations caused by virtual events and real events by judging the source of the user's emotions, and dynamically calculate the user's emotional intensity through multi-modal signals, which are used to record the event content of the real event that causes the user's emotional fluctuation, so as to ensure the authenticity of the recorded content in the process of emotion recognition and exclude false records related to emotional fluctuations caused by virtual events.

[0127] Although the methods described above are illustrated and described as a series of acts for simplicity of explanation, it should be understood and appreciated that the methods are not limited by the order of acts, as some acts may occur in a different order and / or concurrently with other acts not illustrated and described herein or other acts that are understood by those skilled in the art, in accordance with one or more embodiments.

[0128] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0129] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code. The computer-readable medium includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0130] The foregoing description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for recording emotional fluctuation events, characterized in that: The following steps are involved: Obtaining the EEG signal of the user to be recorded; Determining, according to the EEG signal, a probability value for representing that the emotion of the user to be recorded originates from a virtual event; as well as In response to the probability value being less than a preset first threshold, it is determined that the emotional fluctuation of the user to be recorded is caused by a real event, and the event content of the real event is recorded.

2. The recording method according to claim 1, wherein: The electroencephalogram signal includes α wave, β wave, γ wave, δ wave and θ wave. The step of determining the probability value used to characterize that the emotion of the user to be recorded originates from a virtual event according to the electroencephalogram signal includes: Using a filter to remove noise signals from each of the EEG signals, wherein the noise signals at least include power frequency interference and myoelectric noise generated when the power system is working; and Performing frequency domain feature analysis on each of the EEG signals to obtain a power spectrum density of each of the EEG signals, thereby extracting a feature value of each of the EEG signals; and The probability value is determined according to the characteristic values ​​of the α wave, the β wave, the γ wave, the δ wave, and the θ wave.

3. The recording method according to claim 1, wherein: The step of determining, based on the EEG signal, a probability value for characterizing that the emotion of the user to be recorded originates from a virtual event further comprises: Acquiring environmental signals, facial expression signals and physiological signals of the user to be recorded; Preprocessing the environmental signal, the facial expression signal, and the physiological signal to obtain characteristic values ​​of the environmental signal, the facial expression signal, and the physiological signal; and Normalizing the characteristic values ​​of the environmental signal, the facial expression signal, and the physiological signal to map each characteristic value to the same range; and The probability value is determined according to the characteristic values ​​of each of the EEG signal, the environmental signal, the facial expression signal and the physiological signal.

4. The recording method according to claim 3, characterized in that: In response to the probability value being less than a preset first threshold, the step of determining that the emotional fluctuation of the user to be recorded is caused by a real event and recording the event content of the real event includes: Determining an emotion intensity value for characterizing the emotion intensity of the user to be recorded according to the EEG signal, the environmental signal, the facial expression signal, and the physiological signal; and In response to the emotion intensity value being greater than a preset second threshold value, and the fluctuation duration being greater than a preset third threshold value, a first instruction to photograph the user to be recorded is triggered to record the event content of the real event.

5. The recording method according to claim 4, characterized in that The step of determining the emotion intensity value for characterizing the emotion intensity of the user to be recorded according to the EEG signal, the environmental signal, the facial expression signal and the physiological signal comprises: Respectively obtaining initial first weight values ​​of the EEG signal, the environmental signal, the facial expression signal and the physiological signal; According to the variation range of the characteristic values ​​of the EEG signal, the environmental signal, the facial expression signal and / or the physiological signal, each of the first weight values ​​is adjusted to respectively determine the second weight values ​​of the EEG signal, the environmental signal, the facial expression signal and the physiological signal: W i =W i baSe +α·ΔF i Among them, W i base is the first weight value of the i-th signal, W i is the second weight value of the i-th signal, ΔF i is the variation range of the characteristic value of the i-th signal, α is the weight adjustment factor; and Determine the emotion intensity value according to each of the second weight values ​​and the characteristic value of each of the signals: Among them, F i is the eigenvalue of the ith signal.

6. The recording method according to claim 5, characterized in that The step of adjusting the first weight value of the EEG signal according to the change of the EEG signal to determine the second weight value of the EEG signal comprises: Determine the weight adjustment value of the EEG signal according to the weight adjustment factor and the characteristic value of the EEG signal: Where, ΔW EEG is the weight adjustment value of the EEG signal, α is the weight adjustment factor, E EEG is the characteristic value of the EEG signal; and Determine a second weight value of the EEG signal according to the first weight value and the weight adjustment value: in, is the first weight value of the EEG signal, W EEG is the second weight value of the EEG signal.

7. The recording method according to claim 3, characterized in that: The following steps are also included: In response to the probability value being greater than the first threshold, adjusting the emotion intensity value according to the probability value: in, is the adjusted emotion intensity value, I emotion is the emotion intensity value before adjustment, P virtual is the probability value.

8. The recording method according to claim 3, characterized in that: After the first instruction of triggering the shooting of the user to be recorded to record the event content of the real event, the following steps are also included: An emotion fluctuation analysis report of the user to be recorded is automatically generated based on the video data and / or audio data during the period of emotion fluctuation of the user to be recorded, wherein the emotion fluctuation analysis report at least includes the fluctuation duration, peak intensity and occurrence time of the emotion fluctuation.

9. The recording method according to claim 3, characterized in that: After the first instruction of triggering the shooting of the user to be recorded to record the event content of the real event, the following steps are also included: In response to the probability value being greater than the first threshold, or the emotion intensity value being less than the preset fourth threshold and the duration of its fluctuation being less than the third threshold, or the amplitude of change of the signal characteristic values ​​of each of the EEG signal, the environmental signal, the facial expression signal and the physiological signal being lower than the preset fifth threshold, a second instruction to close the shooting of the user to be recorded is triggered.

10. A device for recording emotional fluctuation events, characterized in that: include: A signal acquisition module, used to acquire a variety of signals of the user to be recorded, wherein the signal acquisition module at least includes an EEG sensor, an environmental sensor, a camera and an intelligent wearable device, and the signals at least include EEG signals, environmental signals, facial expression signals and physiological signals; A data processing module, used to determine, based on the plurality of signals, a probability value for characterizing that the emotion of the user to be recorded originates from a virtual event; and The shooting module is used to record the event content of the real event that causes the emotional fluctuation of the user to be recorded.

11. The recording device according to claim 10, characterized in that Also includes: a memory having computer instructions stored thereon; and A controller is connected to the memory, the signal acquisition module, the data processing module and the shooting module, and is used to execute computer instructions stored in the memory to implement the method for recording emotion fluctuation events as described in any one of claims 1 to 9.

12. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method for recording an emotion fluctuation event according to any one of claims 1 to 9 is implemented.

Images