Method, device and storage medium for recording emotional fluctuation events

Abstract

The present invention provides a method for recording an emotion fluctuation event, a device for recording an emotion fluctuation event, and a computer-readable storage medium. The method for recording an emotion fluctuation event comprises the following steps: obtaining an EEG signal of a user to be recorded; determining, based on the EEG signal, a probability value for characterizing that the emotion of the user to be recorded originates from a virtual event; and in response to the probability value being less than a preset first threshold, determining that the emotion fluctuation of the user to be recorded is caused by a real event, and recording the event content of the real event. The present invention can distinguish between emotion fluctuations caused by virtual events and real events by judging the source of the user's emotions, and dynamically calculate the user's emotion intensity through multimodal signals, which is used to record the event content of the real event that caused the user's emotion fluctuation, thereby ensuring the authenticity of the recorded content during the emotion recognition process, and excluding erroneous records of emotion fluctuations caused by virtual events.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, and more particularly to a method, apparatus, and computer-readable storage medium for recording emotional fluctuation events. Prior Technology

[0002] With the rapid development of smart technologies, people's needs for emotional expression and life recording are becoming increasingly refined and personalized. Electroencephalograms (EEGs) are often used to identify emotional states and record events that trigger emotional fluctuations because they directly reflect brain activity. However, existing recording devices mostly focus only on classifying emotions themselves, ignoring the source and intensity changes of emotions. This leads to virtual events (such as watching videos, chatting, and reminiscing) being frequently captured incorrectly, reducing analysis efficiency. To overcome this deficiency, there is an urgent need in this field for an improved method for recording emotional fluctuation events, which can identify and record only emotional fluctuations triggered by real events, ensuring the authenticity of the recorded content and eliminating erroneous recordings caused by virtual events. Summary of the Invention

[0003] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0004] To overcome the aforementioned deficiencies in the existing technology, the present invention provides a method for recording emotional fluctuation events, a device for recording emotional fluctuation events, and a computer-readable storage medium. It can distinguish between emotional fluctuations caused by virtual events and those caused by real events by judging the source of the user's emotions, and dynamically calculate the intensity of the user's emotions through multimodal signals. This is used to record the event content of the real events that caused the user's emotional fluctuations, thereby ensuring the authenticity of the recorded content during the emotion recognition process and eliminating erroneous records of emotional fluctuations caused by virtual events.

[0005] Specifically, the method for recording emotional fluctuation events according to the first aspect of the present invention includes the following steps: acquiring the electroencephalogram (EEG) signal of the user to be recorded; determining, based on the EEG signal, a probability value for characterizing that the user's emotions originate from a virtual event; and when the probability value is less than a preset first threshold, determining that the user's emotional fluctuations are caused by a real event, and recording the event content of the real event.

[0006] Furthermore, in some embodiments of the present invention, the electroencephalogram (EEG) signal includes Wave, Wave, Wave, spread The step of determining, based on the EEG signals, the probability value used to characterize the emotion of the user to be recorded originating from a virtual event includes: using a filter to remove noise signals from each of the EEG signals, wherein the noise signals include at least power frequency interference and electromyographic noise generated during power system operation; and performing frequency domain feature analysis on each of the filtered EEG signals to obtain the power spectral density (PSD) of each EEG signal, thereby extracting feature values ​​of each EEG signal; and based on the... Wave, the Wave, the Wave, the Affected by Multiple characteristic values ​​of the wave; and the probability value determined based on these characteristic values ​​of each of the said electroencephalogram (EEG) signals.

[0007] Furthermore, in some embodiments of the present invention, the step of determining the probability value further includes: acquiring environmental signals, facial expression signals, and physiological signals of the user to be recorded; preprocessing the environmental signals, facial expression signals, and physiological signals to obtain feature values ​​of the environmental signals, facial expression signals, and physiological signals; normalizing the feature values ​​to map each feature value to the same range; and determining the probability value based on the feature values ​​of each of the EEG signals, environmental signals, facial expression signals, and physiological signals.

[0008] Furthermore, in some embodiments of the present invention, the step of recording the event content of the real event includes: determining an emotion intensity value to characterize the emotion intensity of the user to be recorded based on the electroencephalogram (EEG) signal, the environmental signal, the facial expression signal, and the physiological signal; and triggering a first instruction to photograph the user to be recorded when the emotion intensity value is greater than a preset second threshold and its fluctuation duration is greater than a preset third threshold, so as to record the event content of the real event.

[0009] Further, in some embodiments of the present invention, the step of determining the emotion intensity value used to characterize the emotion intensity of the user to be recorded includes: acquiring initial first weight values ​​for the electroencephalogram (EEG) signal, the environmental signal, the facial expression signal, and the physiological signal, respectively; adjusting each of the first weight values ​​according to the change amplitude of the characteristic values ​​of the EEG signal, the environmental signal, the facial expression signal, and / or the physiological signal, so as to determine second weight values ​​for the EEG signal, the environmental signal, the facial expression signal, and the physiological signal, respectively. in, For the first The first weight value of the signal. For the first The second weight value of the signal. For the first The magnitude of change in the characteristic values ​​of the signal. The weighting adjustment factor is used; and the emotion intensity value is determined based on each of the second weight values ​​and the characteristic values ​​of each of the signals. in, For the first The characteristic values ​​of a signal.

[0010] Furthermore, in some embodiments of the present invention, the step of determining each second weight value of the electroencephalogram (EEG) signal based on adjusting each of the first weight values ​​of the EEG signal includes: determining the weight adjustment value of the EEG signal based on the weight adjustment factor and the feature value of the EEG signal. in, The weighting adjustment value for the EEG signal. The weight adjustment factor. The first weight value is a feature value of the EEG signal; and the second weight value of the EEG signal is determined based on the first weight value and the weight adjustment value. in, This is the first weight value of the EEG signal. This is the second weight value of the electroencephalogram (EEG) signal.

[0011] Furthermore, in some embodiments of the present invention, the method for recording emotional fluctuation events further includes the following steps: responding to a probability value greater than or equal to the first threshold, adjusting the emotional intensity value according to the probability value: in, This is the adjusted emotional intensity value. The emotional intensity value before adjustment. Let be the probability value.

[0012] Furthermore, in some embodiments of the present invention, after recording the event content of the real event, the method further includes the following step: automatically generating an emotional fluctuation analysis report of the user to be recorded based on the video and / or audio data during the period of emotional fluctuation. The emotional fluctuation analysis report includes at least the duration of the emotional fluctuation, the peak intensity, and the time of occurrence.

[0013] Furthermore, in some embodiments of the present invention, after recording the event content of the real event, the following steps are also included: when the probability value is greater than or equal to the first threshold, or the emotion intensity value is less than a preset fourth threshold and its fluctuation duration is less than the third threshold, or the change amplitude of the signal characteristic values ​​of each of the EEG signals, the environmental signals, the facial expression signals and the physiological signals is lower than a preset fifth threshold, a second instruction to turn off the recording of the user to be recorded is triggered.

[0014] Furthermore, the recording device for emotional fluctuation events provided by the second aspect of the present invention includes a signal acquisition module, a data processing module, and a shooting module. The signal acquisition module is used to acquire various signals from the user to be recorded. The signal acquisition module includes at least an electroencephalogram (EEG) sensor, an environmental sensor, a camera, and a smart wearable device. The signals include at least EEG signals, environmental signals, facial expression signals, and physiological signals. The data processing module is used to determine, based on the various signals, a probability value representing that the user's emotion originates from a virtual event and an emotion intensity value. The shooting module is used to selectively record the event content of real events that cause emotional fluctuations in the user to be recorded, in response to the determination of the data processing module.

[0015] Furthermore, the computer-readable storage medium provided by the cooperating manufacturer according to the present invention stores computer instructions thereon. When the computer instructions are executed by a processor, the method for recording emotional fluctuation events as provided in the first aspect of the present invention is implemented. Simple Explanation of the Diagram

[0016] The above-described features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of this disclosure in conjunction with the following drawings. In the drawings, the elements are not necessarily drawn to scale, and elements having similar related properties or features may have the same or similar graphic references. Figure 1 shows a schematic diagram of the structure of a recording device for emotional fluctuation events provided according to some embodiments of the present invention. Figure 2 shows a flowchart illustrating a method for recording emotional fluctuation events according to some embodiments of the present invention. Figure 3 shows a flowchart illustrating a method for recording emotional fluctuation events according to some embodiments of the present invention. Implementation

[0017] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention is presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived from the scope of the claims of the present invention. To provide a deep understanding of the invention, many specific details will be included in the following description. The invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0018] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0019] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and figures. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0020] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various elements, regions, layers, and / or portions, these elements, regions, layers, and / or portions should not be limited by these terms, and these terms are only used to distinguish different elements, regions, layers, and / or portions. Therefore, the first elements, regions, layers, and / or portions discussed below may be referred to as second elements, regions, layers, and / or portions without departing from some embodiments of the present invention.

[0021] As mentioned above, with the rapid development of smart technology, people's needs for emotional expression and life recording are becoming increasingly refined and personalized. Electroencephalograms (EEGs), as a direct window reflecting brain activity, are often used to identify emotional states and record events that cause emotional fluctuations in users. However, existing recording devices for emotional events focus on classifying the identified emotions themselves, ignoring the source of the emotion and changes in its intensity. This can lead to the mis-capture of virtual events that don't need to be recorded, such as watching videos, chatting, or reminiscing, thus reducing the efficiency of the emotion recognition and analysis process.

[0022] To overcome the aforementioned deficiencies in the existing technology, the present invention provides a method for recording emotional fluctuation events, a device for recording emotional fluctuation events, and a computer-readable storage medium. It can distinguish between emotional fluctuations caused by virtual events and those caused by real events by judging the source of the user's emotions, and dynamically calculate the intensity of the user's emotions through multimodal signals. This is used to record the event content of the real events that caused the user's emotional fluctuations, thereby ensuring the authenticity of the recorded content during the emotion recognition process and eliminating erroneous records of emotional fluctuations caused by virtual events.

[0023] In some non-limiting embodiments, the method for recording emotional fluctuation events provided in the first aspect of the present invention can be implemented based on the recording device for recording emotional fluctuation events provided in the second aspect of the present invention.

[0024] Please refer to Figure 1 for details. Figure 1 shows a schematic diagram of the structure of a recording device 100 for recording emotional fluctuation events according to some embodiments of the present invention.

[0025] In the embodiment shown in Figure 1, the recording device for emotional fluctuation events provided by the second aspect of the present invention includes a signal acquisition module 11, a data processing module 12, and a shooting module (not shown). Here, the signal acquisition module 11 is used to acquire various signals from the user to be recorded. The signal acquisition module 11 includes at least an EEG sensor 111, an environmental sensor 112, a camera 113, and a smart wearable device 114; the signals include at least EEG signals, environmental signals, facial expression signals, and physiological signals. The data processing module 12 is used to determine, based on the various signals, the probability value representing that the user's emotions originate from a virtual event. The data processing module 12 includes an ESP32 or STM32 processor, an analog-to-digital converter (ADC), and a data fusion processing unit. The shooting module is used to record the event content of real events that cause emotional fluctuations in the user to be recorded.

[0026] Specifically, the EEG sensor 111 can collect the EEG signals of the user from the forehead or other specific areas. The environmental sensor 112 is used to collect environmental signals such as sound and light in the user's surroundings. The camera 113 is used to capture facial expression signals such as facial expressions. The smart wearable device 114 is used to collect physiological signals such as the user's heart rate and skin conductivity.

[0027] Furthermore, in the embodiment shown in Figure 1, the recording device for emotional fluctuation events provided in the second aspect of the present invention may optionally include a power management module 13, a ring-shaped buffer storage module 14, a persistent storage module 15, and a user interface 16. Here, the power management module 13 includes a battery management integrated circuit and a lithium battery. The ring-shaped buffer storage module 14 includes high-speed storage DDR (Double Data Rate) memory, and the persistent storage module 15 includes an eMMC or SD card. The ring-shaped buffer storage module 14 cooperates with the persistent storage module 15 to record event content data acquired by the recording module. The user interface 16 uses wireless communication to obtain an emotional fluctuation analysis report generated based on the event content and displays the analysis report to the user.

[0028] Furthermore, in some non-limiting embodiments, the recording device for emotional fluctuation events provided in the second aspect of the present invention further includes a memory and a controller. Here, the memory includes, but is not limited to, the computer-readable storage medium provided by the aforementioned cooperating manufacturer, on which computer instructions are stored. The controller is connected to the memory, the signal acquisition module 11, the data processing module 12, and the imaging module, and is configured to execute the computer instructions stored in the memory to implement the recording method for emotional fluctuation events as provided in the first aspect of the present invention.

[0029] The operation of the device 100 of this invention is as follows: First, the signal acquisition module 11 (including an EEG sensor 111, etc.) continuously acquires multimodal signals. These signals are transmitted to the data processing module 12. The controller in the data processing module 12 executes the aforementioned algorithm, first determining the probability of the event being real or fake. If it is determined to be a real event and the emotional intensity meets the standard, the controller will issue a first command to the recording module to start recording. The recorded image data can be temporarily stored in the circular buffer storage module 14 and transferred to the persistent storage module 15 after recording is completed. Finally, the data processing module 12 can generate an analysis report and present it to the user through the user interface 16.

[0030] The working principle of the aforementioned recording device for emotional fluctuation events will be described below with reference to embodiments of methods for recording such events. Those skilled in the art will understand that these embodiments of recording methods are merely non-limiting implementations provided by the present invention, intended to clearly demonstrate the main concepts of the invention and provide specific solutions convenient for public implementation, rather than limiting all functions or operating methods of the recording device. Similarly, this recording device for emotional fluctuation events is also only one non-limiting implementation provided by the present invention, and does not limit the executing entity or execution order of the steps in these methods for recording emotional fluctuation events.

[0031] Please refer to Figures 2 and 3 for details. Figure 2 shows a schematic flowchart of a method for recording emotional fluctuation events according to some embodiments of the present invention, process 200. Figure 3 shows a schematic flowchart of a method for recording emotional fluctuation events according to some embodiments of the present invention, process 300.

[0032] As shown in Figures 2 and 3, the recording device provided in the second aspect of the present invention can first acquire the brain signals of the user to be recorded via the EEG sensor 111 in the information acquisition module 11, and then record the acquired data via the circular buffer in the circular buffer storage module 14.

[0033] Specifically, the above-mentioned EEG signals include Wave, Wave, Wave, spread Wave. The frequency range of the wave is 0.5Hz to 4Hz, and it is associated with deep sleep and unconscious states. The frequency range of the wave is 4Hz to 8Hz, and it is associated with memory, imagination, and relaxation. The frequency range of the waves is 8Hz to 12Hz, which reflects the background brainwave activity during relaxation or concentration. The frequency range of the wave is 12Hz to 30Hz, and it is associated with attention, anxiety, and cognitive load. The frequency range of the wave is 30Hz to 100Hz, and it is associated with higher cognitive processing, perception, and emotion integration.

[0034] The recording device can then determine the probability value that the user's emotions are derived from a virtual event based on the EEG signals.

[0035] Furthermore, in some embodiments, the recording device may employ filters to remove noise signals from each electroencephalogram (EEG) signal. Here, the noise signals include at least power frequency interference (e.g., 50Hz power frequency interference) and electromyographic noise generated during power system operation.

[0036] Subsequently, the recording device can perform frequency domain feature analysis on each EEG signal, converting each EEG signal from the time domain to the frequency domain to obtain the power spectral density of each EEG signal, thereby extracting the feature values ​​of each EEG signal, which can then be used to represent the physiological and psychological information carried by each EEG wave at different frequencies in the EEG signal.

[0037] After that, the recording device can... Wave, Wave, Wave, spread The eigenvalues ​​of the wave determine the probability value.

[0038] Specifically, emotional fluctuations triggered by virtual events such as making phone calls, watching entertainment programs, or chatting with others typically stimulate brain activity in brain regions related to internal thinking and imagination, reflected in electroencephalogram (EEG) signals. Increased activity in alpha waves (4-8Hz) and alpha waves (8-12Hz), while relatively subdued activity in gamma waves (30-100Hz), associated with high-intensity external perception. Conversely, emotional fluctuations caused by real events typically involve direct, high-intensity sensory input and immediate responses to the external environment, reflected in EEG signals as significantly increased activity in gamma and beta waves (12-30Hz), with large amplitude and long duration.

[0039] Thus, the recording device provided in the second aspect of the present invention can distinguish whether the source of emotional fluctuations is a real event or a virtual event based solely on the characteristic values ​​of various brain waves in the electroencephalogram (EEG) signal.

[0040] In addition, as shown in Figure 3, the recording device can also acquire the environmental signals, facial expression signals and physiological signals of the user to be recorded through the environmental sensor 112, camera 113 and smart wearable device 114 in the information acquisition module 11.

[0041] The recording device can then preprocess the environmental signals, facial expression signals, and physiological signals to obtain their characteristic values.

[0042] Specifically, the recording device can use a microphone to collect sound signals around the user being recorded and perform spectral analysis (FFT, Fast Fourier Transform) on them to extract features such as sound intensity and frequency analysis of the sound signals.

[0043] In addition, the recording device can use computer vision image processing libraries such as OpenCV or Dlib to identify facial feature points of the user to be recorded, so as to calculate facial expression features such as the smile index and the degree of eyebrow raising.

[0044] In addition, the recording device can use smartwatches, smart bracelets, etc. to collect information such as heart rate and skin conductivity, which can be used to help analyze emotional fluctuations.

[0045] Then, the recording device can normalize the feature values ​​of the above-mentioned environmental signals, facial expression signals and physiological signals to map each feature value to the same range (e.g., [0,1]).

[0046] Then, the recording device can determine the probability value based on the characteristic values ​​of each EEG signal, environmental signal, facial expression signal, and physiological signal.

[0047] Please refer to Table 1 for details. Table 1 shows the characteristic differences between various signals generated by emotional fluctuations caused by virtual events and real events, provided by some embodiments of the present invention. Table 1. Characteristic differences of various signals generated by emotional fluctuations caused by virtual events and real events. category Emotional fluctuations caused by virtual events Emotional fluctuations caused by real events EEG signal characteristics In EEG signals Waves dominate, accompanied by slight fluctuations. The overall fluctuation is relatively low. In EEG signals wave and The wave is significantly enhanced, with large amplitude and long duration. Environmental signal characteristics The ambient light and sound changes are relatively stable, and the frequency characteristics are synchronized with the virtual content (such as movie sound effects). Significant changes in environmental signals, such as sudden noise, strong fluctuations in light, or realistic human voices. Facial expression signal characteristics The frequency of facial expressions is relatively high, but the intensity is relatively weak (such as a slight laugh while watching a funny video or a slight furrowing of the brow in a tense scene). The facial expressions change dramatically, accompanied by large facial movements such as widening eyes and opening the mouth, indicating tension, surprise, or fear. Physiological signal characteristics Users typically maintain a stable posture with minimal movement (such as sitting while watching videos or using virtual interactive tools). Users may exhibit subconscious actions (such as taking a step back, quickly turning their head, or clenching their fists).

[0048] Thus, the recording device provided in the second aspect of the present invention can further combine the feature values ​​of other multi-source signals to distinguish whether the source of emotional fluctuations is a real event or a virtual event, thereby improving the accuracy of identifying the source of emotions.

[0049] In a preferred embodiment, the probability value is determined by a pre-trained machine learning classifier. The training process of the classifier may include: (1) Data collection and labeling: Invite multiple subjects to experience real events (such as watching a sudden fright video) and virtual events (such as watching a landscape video or recalling events with eyes closed), and simultaneously collect their multimodal signals, manually labeling the corresponding data segments as 'real' or 'virtual'; (2) Feature engineering: Extract feature values ​​from each signal (such as the power spectral density of each EEG segment, heart rate variability, etc.); (3) Model training: Input the labeled feature data into a classification model (e.g., Support Vector Machine (SVM), Logistic Regression, or a lightweight neural network) for training, so that it learns the data patterns of real and virtual events. After training, the model can receive real-time signal feature values ​​online and output a probability value between the two, which is the probability value that the emotion originates from the virtual event.

[0050] Subsequently, if the probability value is less than a preset first threshold (e.g., 0.2), the recording device can determine that the emotional fluctuation of the user to be recorded is caused by a real event and record the content of the real event.

[0051] Furthermore, as shown in Figure 3, the recording device can determine the emotion intensity value used to characterize the emotion intensity of the user being recorded based on EEG signals, environmental signals, facial expression signals, and physiological signals.

[0052] Specifically, the recording device can acquire initial weight values ​​for electroencephalogram (EEG) signals, environmental signals, facial expression signals, and physiological signals respectively. Here, the recording device can set a higher initial weight value for EEG signals (e.g., 0.6) and lower weight values ​​for other signals (e.g., 0.2 for environmental signals, 0.1 for facial expression signals, and 0.1 for physiological signals).

[0053] Furthermore, in determining the initial weight value of the signal, the recording device can collect labeled data on various signals and emotional changes in multiple scenarios. These labels include the actual contribution of each signal to the emotional judgment. Subsequently, the recording device can use statistical methods to analyze the strong correlations between different signals. For example, in a movie-watching scenario, EEG signals show a higher correlation with emotional fluctuations, while during skydiving, abrupt changes in physiological signals contribute significantly.

[0054] Furthermore, in some preferred embodiments, the recording device can be trained based on machine learning models such as multimodal fusion networks and regression analysis to map the contribution of different signals to emotion judgment.

[0055] Furthermore, in some alternative embodiments, in the absence of sufficient training data, the recording device may initially set a first weight value based on prior knowledge of the importance of the signals, such as the fact that EEG typically dominates in emotion judgment. Subsequently, in dynamic scenarios, the weights of physiological and environmental signals should be increased. Conversely, in static scenarios, the weights of EEG signals and facial expression signals should be increased.

[0056] Subsequently, in the embodiment shown in Figure 3, the recording device can adjust the first weight values ​​based on the variation amplitude of the characteristic values ​​of the electroencephalogram (EEG) signal, environmental signal, facial expression signal, and / or physiological signal, so as to determine the second weight values ​​of the EEG signal, environmental signal, facial expression signal, and physiological signal respectively:

[0057] in, For the first The first weight value of the signal, For the first The second weight value of the signal, For the first The variation range of the signal characteristic values This is the weighting adjustment factor.

[0058] For example, the recording device can increase the weight of EEG signals when the user's internal emotional signals fluctuate dramatically. It can also increase the weight of environmental signals when interference is amplified, such as noise or frequent changes in light. Furthermore, it can increase the weight of facial expression signals when there are significant changes in facial expression, such as a large raising of the eyebrows.

[0059] Furthermore, in determining the second weight value of the EEG signal, the recording device can determine the weight adjustment value of the EEG signal based on the weight adjustment factor and the characteristic values ​​of the EEG signal.

[0060] in, The weighted adjustment value for the EEG signal. For weight adjustment factors, These are the characteristic values ​​of the electroencephalogram (EEG) signal.

[0061] Then, the device can determine the second weight value of the EEG signal based on the first weight value and the weight adjustment value:

[0062] in, The first weighting value of the EEG signal. This is the second weighting value of the EEG signal.

[0063] Then, the recording device can input the second weight values ​​and the feature values ​​of each signal into a pre-trained deep learning model (e.g., an LSTM model or a Transformer model) to determine the emotion intensity value.

[0064] in, For the first The characteristic values ​​of a signal.

[0065] Subsequently, if the emotional intensity value is greater than the preset second threshold and the duration of its fluctuation is greater than the preset third threshold (e.g., 3s, 30s, 1min), the recording device can trigger the first instruction to record the user to be recorded, so as to record the content of the real event.

[0066] Here, the second threshold used to trigger emotion intensity is set in the range of [0.6, 0.9]. The larger the second threshold, the greater the intensity of the user's emotion required for the recording device to trigger the shooting module to record the event, and the more intense the recorded user's emotional fluctuations.

[0067] Furthermore, in some preferred embodiments, when the probability value is greater than or equal to a first threshold, the recording device can also adjust the emotion intensity value based on the probability value.

[0068] in, This is the adjusted emotional intensity value. The emotional intensity value before adjustment. This represents the probability of virtual emotions.

[0069] Thus, the recording device provided in the second aspect of the present invention can dynamically adjust the intensity of emotions, so that when the probability value of the emotional fluctuation originating from the virtual event is high, the corresponding intensity value of the emotion will decrease, thereby avoiding the accidental triggering of the shooting module to record the event content of the virtual event.

[0070] Furthermore, as shown in Figure 3, in response to the conditions for stopping recording being met, the recording device can trigger a second instruction to stop recording the user being recorded and store the recorded content during the recording period in the persistent storage module 15.

[0071] Furthermore, in some embodiments, in response to a probability value greater than or equal to a first threshold, the recording device can trigger a second instruction to stop recording the user being recorded. Here, after determining that the emotional fluctuation originates from a virtual event, the recording device does not start or stop the recording module and enters a low-power mode, retaining only the emotion detection function.

[0072] Alternatively, in some embodiments, in response to an emotional intensity value being less than a preset fourth threshold and its fluctuation duration being less than a third threshold, or the change amplitude of the signal characteristic values ​​of each EEG signal, environmental signal, facial expression signal, and physiological signal being less than a preset fifth threshold, a second instruction to stop recording the user is triggered. Here, the recording device does not start or stop the recording module after the emotional fluctuation intensity is not significant or the emotion tends to stabilize, and automatically saves the recorded content.

[0073] Next, as shown in Figure 3, the recording device can automatically generate an emotional fluctuation analysis report of the user to be recorded based on the video and / or audio data during the period of the user's emotional fluctuation, and display it to the user through the user interface 16. Here, the emotional fluctuation analysis report includes at least the duration of the emotional fluctuation, the peak intensity, and the time of occurrence.

[0074] Thus, the recording device provided in the second aspect of the present invention can utilize a passive brain-computer interface (BCI) to automatically record complete events in life that induce strong emotional fluctuations in the user without the user consciously triggering the recording. This is achieved by collecting and analyzing the unconscious brain signals of the user, thereby reducing the burden of use.

[0075] For example, in the field of psychotherapy support, recording devices can capture triggering events in a patient's life that cause various emotional fluctuations, providing a basis for psychological assessment.

[0076] For example, in the field of criminal investigation, due to the special nature of the crime scene, eyewitnesses may find it difficult to promptly and proactively record crucial evidence at the crime scene using traditional devices (e.g., mobile phone cameras). Furthermore, eyewitnesses often experience significant emotional fluctuations such as surprise, fear, and tension when witnessing sudden and unusual criminal acts. Therefore, the recording device provided in the second aspect of this invention can automatically start recording based on emotional fluctuations in emergency situations, without requiring active user operation, ensuring the confidentiality, timeliness, and integrity of evidence collection, thereby providing support for case investigation.

[0077] For example, in the field of security monitoring, due to the special nature of extreme sports such as rock climbing, skydiving, skiing, and diving, participants cannot operate camera equipment to record in real time. Therefore, the recording device provided in the second aspect of the present invention can automatically record valuable images based on strong emotional fluctuations such as fear, excitement, and tension. In addition, if participants face sudden mistakes or dangers, and their emotional fluctuations, such as persistently high levels of fear and tension signals indicating a dangerous situation, show signs of danger, the recording device can transmit the recorded images to the ground safety team via a communication device, facilitating the rescue team's understanding of the dangerous situation and timely provision of appropriate rescue.

[0078] To verify the accuracy of the method, apparatus, and storage medium for recording emotional fluctuation events provided by this invention in analyzing the source of emotions, technicians can conduct tests using both virtual and real events that cause emotional fluctuations in users.

[0079] For example, a user wearing the emotional fluctuation recording device provided in the second aspect of the present invention is watching a movie. The recording device can collect the user's electroencephalogram (EEG) signals, environmental signals, facial expression signals, and physiological signals to analyze the source of the user's emotional fluctuations.

[0080] Here, in the user's brain signals, Wave, spread Waves dominate, The activity level of the waves is low. In the user's environmental signals, the sound signals are stable background noises such as movie dialogue and sound effects, with continuous and consistent frequency characteristics. The light signals are regular changes in light emitted from the screen, without sudden flickering or significant interference. The user's facial expression signals show a small overall amplitude and slow changes. The user's physiological signals indicate that the user is in a static state, with their eyes continuously focused on the screen.

[0081] Thus, by combining the characteristic values ​​of the aforementioned signal, the recording device can determine that the probability value of the emotional fluctuation originating from the virtual event is 0.85, which is much greater than the preset first threshold (e.g., 0.2). Therefore, the recording device can determine that the emotional fluctuation was caused by the virtual event.

[0082] The recording device then sets the first weight value of the EEG signal to 0.6, the first weight value of the environmental signal to 0.3, and the sum of the first weight values ​​of the facial expression signal and the physiological signal to 0.1. Next, the recording device adjusts the above signals, setting the second weight value of the EEG signal to 0.4, the second weight value of the environmental signal to 0.5, and the sum of the second weight values ​​of the facial expression signal and the physiological signal to 0.1.

[0083] Then, the recording device can calculate the intensity of the emotion:

[0084] If the probability that the emotional fluctuation originates from a virtual event is greater than the first threshold, adjust the emotional intensity value:

[0085] Thus, if the probability value of an emotional fluctuation originating from a virtual event is greater than a first threshold and the emotional intensity value is less than a second threshold (e.g., 0.7), the shooting module is not triggered, and the event content of the virtual event causing the user's emotional fluctuation is not recorded, thereby avoiding unnecessary recording and ensuring storage efficiency and privacy protection. This process achieves an efficient combination of emotion source classification and shooting logic, highlighting the intelligence and applicability of the emotional fluctuation event recording device provided in the second aspect of the present invention.

[0086] For example, a user wearing the emotional fluctuation recording device provided in the second aspect of the present invention participates in a skydiving activity. The recording device can collect the user's electroencephalogram (EEG) signals, environmental signals, facial expression signals, and physiological signals to analyze the source of the user's emotional fluctuations.

[0087] Here, in the user's brain signals, spread Waves dominate, spread Wave activity decreased significantly, indicating the user was in a state of high tension and excitement. In the user's environmental signals, the sound signal showed a sharp increase in wind noise with an irregular spectrum, accompanied by a brief period of silence at the moment of jumping out of the cabin. The light signal showed significant changes in light intensity as the user transitioned from a dark area inside the cabin to a bright area in the air. The user's facial expression signals showed intense facial muscle activity and wide-open eyes. In the user's physiological signals, the IMU sensor detected a significant change in acceleration at the moment of jumping, and the user's attitude angle adjusted rapidly.

[0088] Thus, by combining the characteristic values ​​of the aforementioned signal, the recording device can determine that the probability value of the emotional fluctuation originating from the virtual event is 0.15, which is less than a pre-set first threshold (e.g., 0.2). Therefore, the recording device can determine that the emotional fluctuation was caused by a real event.

[0089] The recording device then sets the first weight value of the EEG signal to 0.4, the first weight value of the environmental signal to 0.3, and the sum of the first weight values ​​of the facial expression signal and the physiological signal to 0.3. Next, the recording device adjusts the above signals, resulting in a second weight value of 0.4 for the EEG signal, a second weight value of 0.3 for the environmental signal, and a sum of the second weight values ​​of the facial expression signal and the physiological signal to 0.3.

[0090] Then, the recording device can calculate the intensity of the emotion:

[0091] Thus, when the probability value of the emotional fluctuation originating from the virtual event is less than the first threshold and the emotional intensity value is greater than the second threshold (e.g., 0.7), the shooting module is triggered to start and record the event content of the real event that caused the user's emotional fluctuation, recording the key moment of the user's skydive. This process highlights the intelligent triggering capability of the emotional fluctuation event recording device provided by the second aspect of the present invention in real high-dynamic scenarios, providing users with valuable images while ensuring the efficient operation of the device.

[0092] In summary, the method, device, and storage medium for recording emotional fluctuation events provided by this invention can distinguish between emotional fluctuations caused by virtual events and those caused by real events by judging the source of the user's emotions, and dynamically calculate the intensity of the user's emotions through multimodal signals to record the event content of the real events that caused the user's emotional fluctuations, thereby ensuring the authenticity of the recorded content during the emotion recognition process and eliminating erroneous records of emotional fluctuations caused by virtual events.

[0093] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0094] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0095] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or via a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then those coaxial cables, fiber optic cables, twisted pairs, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0096] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0097] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0098] 100: Device 200: Process 300: Process

[0099] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none

Claims

1. A method for recording emotional fluctuation events, comprising the following steps: acquiring an electroencephalogram (EEG) signal, an environmental signal, a facial expression signal, and a physiological signal of a user to be recorded; determining a probability value, based on the EEG signal, that the user's emotion originates from a virtual event; and determining that the user's emotional fluctuation is caused by a real event when the probability value is less than a preset first threshold; determining an emotion intensity value, based on the EEG signal, the environmental signal, the facial expression signal, and the physiological signal, that characterizes the intensity of the user's emotion; and triggering a first instruction to photograph the user to record the event content of the real event when the emotion intensity value is greater than a preset second threshold and the duration of the fluctuation is greater than a preset third threshold.

2. The recording method as described in claim 1, wherein the electroencephalogram (EEG) signal includes waves, waves, waves, waves, and waves, and the step of determining, based on the EEG signal, a probability value representing that the emotion of the user to be recorded originates from a virtual event includes: The method involves using filters to remove noise signals from each of the stated EEG signals, wherein the noise signals include at least power frequency interference and electromyographic noise generated during power system operation; performing frequency domain feature analysis on each of the filtered EEG signals to obtain the power spectral density of each EEG signal, thereby extracting a plurality of feature values ​​corresponding to the wave, the wave, the wave, the wave, and the wave; and determining the probability value based on these feature values ​​of each of the stated EEG signals.

3. The recording method as described in claim 1, wherein the step of determining the probability value further includes: The environmental signal, the facial expression signal, and the physiological signal are preprocessed to obtain a plurality of feature values ​​for the environmental signal, the facial expression signal, and the physiological signal; The feature values ​​are normalized to map each feature value to the same range; and the probability value is determined based on the feature values ​​of the EEG signal, the environmental signal, the facial expression signal, and the physiological signal.

4. The recording method as described in claim 3, wherein the step of determining an emotion intensity value to characterize the emotion intensity of the user to be recorded includes: The initial multiple first weight values ​​of the electroencephalogram (EEG) signal, the environmental signal, the facial expression signal, and the physiological signal are obtained respectively. Based on the variation amplitude of the EEG signal, the environmental signal, the facial expression signal, and / or the physiological signal feature value, adjust each of the first weight values ​​to determine a plurality of second weight values ​​for the EEG signal, the environmental signal, the facial expression signal, and the physiological signal respectively: where is the first weight value of the i-th signal, are the second weight values ​​of the i-th signal, is the variation amplitude of the feature value of the i-th signal, and is the weight adjustment factor; and based on each of the second weight values ​​and the feature value of each of the signals, determine the emotion intensity value: where is the feature value of the i-th signal.

5. The recording method as claimed in claim 4, wherein the step of adjusting each of the first weight values ​​of the electroencephalogram (EEG) signals to determine each of the second weight values ​​of the EEG signals comprises: Based on the weight adjustment factor and the characteristic values ​​of the EEG signal, a weight adjustment value for the EEG signal is determined: where is the weight adjustment value of the EEG signal, is the weight adjustment factor, and is the characteristic value of the EEG signal; and based on the first weight value and the weight adjustment value, the second weight values ​​of the EEG signal are determined: where is the first weight value of the EEG signal, and is the second weight value of the EEG signal.

6. The recording method as described in claim 3 further includes the following steps: In response to the probability value being greater than or equal to the first threshold, adjusting the emotion intensity value according to the probability value: wherein, Here, is the adjusted emotional intensity value, is the original emotional intensity value, and is the probability value.

7. The recording method as described in claim 3, wherein after recording the event content of the real event, it further includes the following step: automatically generating an emotion fluctuation analysis report of the user to be recorded based on video and / or audio data during the period of the user's emotional fluctuation, wherein, The emotion fluctuation analysis report shall include at least the duration of the emotion fluctuation, the peak intensity, and the time of occurrence.

8. The recording method as described in claim 3, wherein after recording the event content of the real event, the method further includes the following steps: when the probability value is greater than or equal to the first threshold, or the emotion intensity value is less than a preset fourth threshold and its fluctuation duration is less than the third threshold, or the change amplitude of the signal characteristic values ​​of each of the EEG signals, the environmental signals, the facial expression signals and the physiological signals is lower than a preset fifth threshold, a second instruction to stop recording the user to be recorded is triggered.

9. A recording apparatus for implementing the method described in any one of claims 1 to 8, comprising: A signal acquisition module is used to acquire a plurality of multimodal signals of a user to be recorded, wherein the signal acquisition module includes at least an EEG sensor, an environmental sensor, a camera, and a smart wearable device, and the multimodal signals include at least EEG signals, environmental signals, facial expression signals, and physiological signals; a data processing module is used to determine, based on the multimodal signals, a probability value and an emotion intensity value for characterizing the user's emotions as originating from virtual events; and a recording module is used to selectively record the event content of real events that cause emotional fluctuations in the user to be recorded in response to the judgment of the data processing module.

10. A computer-readable storage medium storing computer instructions that, when executed by a processor, can implement a method for recording emotional fluctuation events as described in any one of claims 1 to 8.

Images