Multimodal Fusion-Based Emotional Tactile Regulation System and Method

Abstract

The present invention discloses an emotional tactile regulation system and method based on multi-modal fusion, including a tactile optimal parameter adjustment module, a tactile generation module, an audiovisual generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module, and a multi-modal fusion emotion recognition module. The system collects various physiological signals of users, fuses the multi-physiological signal features with audio and tactile modal features, combines advanced data processing and analysis techniques, accurately identifies the current emotional state of users in real time, automatically finds tactile parameters by means of optimization theory, and actively regulates the emotional state by applying tactile stimuli to users; it overcomes the limitations of traditional methods such as subjective scales, effectively reduces the influence of unstable physiological signals on the emotion recognition results, and significantly improves the accuracy of emotion detection in the emotional tactile regulation system. The emotional tactile regulation system has broad application potential in the fields of personal emotion management, medical rehabilitation, audio-visual entertainment, etc.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion regulation, and particularly relates to an emotion tactile regulation system and method based on multi-modal fusion. Background Art

[0002] In recent years, the rapid development of emotion computing and tactile technology has given rise to a new field: emotion touch. Emotion computing is dedicated to revealing the mechanisms of emotion generation and expression, while tactile technology focuses on simulating the tactile perception process of humans. The field of emotion touch, which combines emotion information with tactile technology, aims to explore the possibility of using tactile technology in the processes of emotion detection, display, and communication, providing new possibilities for human-computer interaction.

[0003] As the core technology in the field of emotion touch, the emotion tactile regulation system aims to perceive the emotional state of an individual in real time and guide it through tactile stimulation, thereby achieving active regulation of emotions during the interaction process. In the fields of human-computer interaction such as medical rehabilitation and audio-visual entertainment, the emotion tactile regulation system has remarkable application prospects. For example, in the medical field, the system assists in emotion regulation through tactile stimulation, providing innovative means for the treatment of emotional disorders such as depression. In the field of audio-visual entertainment, the system enhances emotional immersion through tactile stimulation, creating a more immersive experience for users.

[0004] However, despite the broad prospects of the emotion tactile regulation system, there are still some technical problems that need to be solved urgently:

[0005] 1. Lack of objective and real-time emotion detection means. The existing emotion detection methods in the emotion tactile regulation system often rely on subjective evaluation means such as scales. This method is easily affected by an individual's subjective consciousness and the external environment, resulting in limited objectivity and real-time nature of the emotional state.

[0006] 2. The influence mechanism between tactile stimulation and emotional state is not yet clear. Although there is a close relationship between touch and emotion, there is still a lack of clear guidance on how to adjust specific tactile parameters to achieve specific emotion regulation, resulting in the system often being difficult to achieve the expected emotion regulation effect in practical applications.

[0007] The above two technical problems restrict the further development of the emotion tactile regulation system. Summary of the Invention

[0008] To solve the above problems, the present invention discloses an emotional tactile regulation system and method based on multimodal fusion. The system collects various physiological signals of the user, such as electroencephalogram and electrocardiogram, etc., fuses the multi-physiological signal features with audio and tactile modality features, combines advanced data processing and analysis technologies, accurately identifies the user's current emotional state in real time, and automatically searches for tactile parameters by means of optimization theory, and realizes the active regulation of the emotional state by applying tactile stimuli to the user.

[0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] An emotional tactile regulation system based on multimodal fusion includes a tactile optimal parameter adjustment module, a tactile generation module, an audiovisual generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module, and a multimodal fusion emotion recognition module. The tactile optimal parameter adjustment module automatically solves the optimal tactile parameters according to the difference between the user's current emotion and the target emotion, and sends them to the tactile generation module to generate a tactile effect. The tactile generation module and the audiovisual generation module cooperate to generate an audiovisual tactile fusion stimulus acting on the user to regulate the user's emotion. The multi-physiological signal acquisition module and the multi-sensory signal acquisition module collect various physiological signals, audio signals, and tactile vibration signals of the user in real time, and input them to the multimodal fusion emotion recognition module to detect the user's current emotional state, and feedback it to the tactile optimal parameter adjustment module to form a closed-loop emotional regulation system.

[0011] The tactile optimal parameter adjustment module, as the core of the emotional tactile regulation system based on multimodal fusion, automatically searches for tactile parameters according to the difference between the user's current emotional state and the target emotion by means of optimization theory, and sends the parameters to the tactile generation module to ensure the effectiveness of emotional regulation. The tactile optimal parameter adjustment module includes a tactile parameter optimization model and a tactile parameter solving module. The tactile parameter optimization model is expressed as

[0012]

[0013] s.t. 0≤P≤P m

[0014] 0≤f≤f m

[0015] 0≤q≤q m

[0016] 0≤r≤r m

[0017] Wherein, M i is the actual power value of a certain electrode i on the electroencephalogram topographic map, M bi is the reference power value of a certain electrode i on the electroencephalogram topographic map, S is the calculated emotional state value, Sb is the target emotional state value, P is the actual power value consumed by the tactile generation module, P m is the maximum power value set for the tactile generation module, f m is the set maximum tactile vibration frequency, q m is the set maximum tactile vibration intensity, r m is the set maximum tactile vibration rhythm, and γ, μ, and φ are the weight coefficients of the model.

[0018] The tactile parameter solving module uses machine learning algorithms such as particle swarm optimization algorithm and reinforcement learning to solve the four parameters in the optimal tactile parameter model, namely the tactile vibration frequency f, the tactile vibration intensity q, the tactile vibration rhythm r, and the tactile vibration position c, and sends the solved parameters to the tactile generation module.

[0019] The tactile generation module is a type of wearable device that expresses touch through vibration, including a vibrating vest, a vibrating bracelet, a vibrating glove, etc. By setting the vibration frequency, intensity, rhythm, and position of the tactile generation device, a specific tactile experience is conveyed. The feature of the tactile generation module is that there is always a background tactile vibration that adapts to the audio, and at the same time, another tactile expression can be achieved based on the four optimal parameters calculated by the tactile parameter optimization model. This tactile expression works in coordination with the background tactile vibration to effectively enhance the user's emotional experience.

[0020] The visual and auditory generation module provides visual and auditory stimuli for the user, including movie clip materials of different emotional types. These audio-visual stimulus materials help guide the user into a specific emotional state, and the audio of the movie clip is used as the basis for the change of the background tactile vibration.

[0021] The multi-physiological signal acquisition module real-time acquires various physiological signals of the user, including 64-channel electroencephalogram (EEG) signals and electrocardiogram (ECG) signals. Among them, the EEG signals are acquired by the EEG signal acquisition module, and the EEG signal acquisition module consists of a 64-lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier. The ECG signals are acquired by the ECG signal acquisition module, and the ECG signal acquisition module is an ActiveTwo series high-lead ECG acquisition system of Biosemi company.

[0022] The multi-sensory signal acquisition module includes an auditory signal acquisition module and a tactile signal acquisition module, which can respectively real-time acquire the audio signal in the visual and auditory generation module and the tactile vibration signal in the tactile generation module.

[0023] The multi-modal fusion emotion recognition module can analyze, process, and recognize the user's current emotional state based on the user's multi-physiological signals and the multi-sensory signals of the induced materials, and then send this emotional state signal to the tactile optimal parameter adjustment module for intelligent regulation of tactile parameters. The multi-modal fusion emotion recognition module includes a signal preprocessing module, a feature extraction module, a feature fusion module, and an emotion decoding module. The signal preprocessing module preprocesses the collected electroencephalogram and electrocardiogram signals, including downsampling, filtering, artifact removal, etc. The feature extraction module extracts features from the auditory signal, tactile signal, preprocessed electroencephalogram, and electrocardiogram signals respectively. The feature fusion module uses a feature fusion algorithm to fuse the multi-physiological signal features with the audio features extracted from the audio signal and the vibration features extracted from the tactile vibration signal. The emotion decoding module uses a classification algorithm to classify the multi-modal fusion features to obtain the user's current emotional state. The superiority of the multi-modal fusion emotion recognition module lies in the fusion of multi-physiological signal features with audio and tactile modality features, which can effectively reduce the impact of unstable physiological signals on the emotion recognition result.

[0024] The beneficial effects of the present invention include:

[0025] 1. The emotion-tactile regulation system of the present invention adopts an emotion detection method based on multi-modal fusion, which fuses multi-physiological signal features with audio and tactile modality features to achieve objective and real-time emotion recognition, overcomes the limitations of traditional subjective scale methods, effectively reduces the impact of unstable physiological signals on the emotion recognition result, and significantly improves the accuracy of emotion detection in the emotion-tactile regulation system.

[0026] 2. The tactile optimal parameter adjustment module in the present invention analyzes the difference between the real-time emotional state and the target emotion, and automatically searches for appropriate tactile parameters by means of optimization theory, so that the tactile stimulation can more accurately guide and regulate the user's emotional state, effectively improving the effect and efficiency of the emotion-tactile regulation system.

[0027] 3. The emotion-tactile regulation system based on multi-modal fusion in the present invention can establish an emotion-tactile database of users, and use big data and large model learning technologies to generate personalized tactile patterns to present customized emotional experiences. Description of the Drawings

[0028] Figure 1 is the principle block diagram of the emotion-tactile regulation system based on multi-modal fusion of the present invention;

[0029] Figure 2 is the schematic diagram of the tactile optimal parameter adjustment module of the present invention;

[0030] Figure 3 is the schematic diagram of the tactile generation module of the present invention;

[0031] Figure 4 Schematic diagram of the multi - physiological signal acquisition module of the present invention;

[0032] Figure 5 Schematic diagram of the multi - sensory signal acquisition module of the present invention;

[0033] Figure 6 Schematic diagram of the multi - modal fusion emotion recognition module of the present invention;

[0034] Figure 7 Flow chart of the emotion - tactile regulation method based on multi - modal fusion of the present invention;

[0035] Figure 8 Experimental paradigm diagram of the tactile emotion regulation system of the present invention.

[0036] List of attached drawing reference signs:

[0037] 1. Tactile optimal parameter adjustment module; 2. Tactile generation module; 3. Visual and auditory generation module; 4. Multi - physiological signal acquisition module; 5. Multi - sensory signal acquisition module; 6. Multi - modal fusion emotion recognition module; 7. Tactile parameter optimization model; 8. Tactile parameter solution module; 9. EEG signal acquisition module; 10. ECG signal acquisition module; 11. Auditory signal acquisition module; 12. Tactile signal acquisition module; 13. Signal pre - processing module; 14. Feature extraction module; 15. Feature fusion module; 16. Emotion decoding module. Detailed implementation manners

[0038] The following further clarifies the present invention in conjunction with the attached drawings and detailed implementation manners. It should be understood that the following detailed implementation manners are only used to illustrate the present invention and not to limit the scope of the present invention.

[0039] Example 1:

[0040] This example describes an emotion - tactile regulation system based on multi - modal fusion, and its overall schematic diagram is as shown in Figure 1As shown in the figure, it includes a tactile optimal parameter adjustment module 1, a tactile generation module 2, an audiovisual generation module 3, a multi-physiological signal acquisition module 4, a multi-sensory signal acquisition module 5, and a multi-modal fusion emotion recognition module 6. The tactile optimal parameter adjustment module 1 automatically solves the optimal tactile parameters according to the difference between the user's current emotion and the target emotion, sends them to the tactile generation module 2 and generates a tactile effect. The tactile generation module 2 and the audiovisual generation module 3 cooperate to generate an audiovisual-tactile fusion stimulus acting on the user to regulate the user's emotion. The multi-physiological signal acquisition module 4 and the multi-sensory signal acquisition module 5 collect various physiological signals, audio signals, and tactile vibration signals of the user in real time, input them into the multi-modal fusion emotion recognition module 6 to detect the user's current emotional state, and feedback them to the tactile optimal parameter adjustment module 1, constituting a closed-loop emotion tactile regulation system based on multi-modal fusion.

[0041] Referring to Figure 2 , Figure 2 is a schematic diagram of the tactile optimal parameter adjustment module 1. The tactile optimal parameter adjustment module 1, as the core of the emotion tactile regulation system based on multi-modal fusion, automatically seeks tactile parameters according to the difference between the user's current emotional state and the target emotion with the help of optimization theory, and sends the parameters to the tactile generation module 2 to ensure the effectiveness of emotion regulation. The tactile optimal parameter adjustment module 1 includes a tactile parameter optimization model 7 and a tactile parameter solution module 8.

[0042] The tactile parameter optimization model 7 establishes a reliable mathematical model by integrating elements such as electroencephalogram signals, emotional states, and vibration power consumption, and is used to adaptively adjust four key parameters in the tactile generation module 2, including tactile vibration frequency f, tactile vibration intensity q, tactile vibration rhythm r, and tactile vibration position c. The tactile parameter optimization model 7 is expressed as

[0043]

[0044] 0 ≤ f ≤ f m

[0045] 0 ≤ q ≤ q m

[0046] 0 ≤ r ≤ r m

[0047] where M i is the actual power value of a certain electrode i on the electroencephalogram topographic map, M bi is the reference power value of a certain electrode i on the electroencephalogram topographic map, S is the calculated emotional state value, S b is the target emotional state value, P is the actual power value consumed by the tactile generation module, P m is the set maximum power value consumed by the tactile generation module, fm is to set the maximum tactile vibration frequency, q m is to set the maximum tactile vibration intensity, r m is to set the maximum tactile vibration rhythm, γ, μ, and are the weight coefficients of the model.

[0048] The tactile parameter solving module 8 includes methods such as optimization algorithms, machine learning, and reinforcement learning, and is used to solve the four parameters in the tactile parameter optimization model 7, namely the tactile vibration frequency f, the tactile vibration intensity q, the tactile vibration rhythm r, and the tactile vibration position c, and send the solved parameters to the tactile generation module 2.

[0049] Referring to Figure 3 , Figure 3 , is a schematic diagram of the tactile generation module 2. The tactile generation module 2 is a type of wearable device that expresses touch through vibration, including a vibrating vest, a vibrating bracelet, a vibrating glove, etc. By setting the vibration frequency, intensity, rhythm, and position of the tactile generation device, a specific tactile experience is transmitted to the user. The characteristic of the tactile generation module 2 is that there is always a background tactile vibration that adapts to the audio, and at the same time, another tactile expression can be realized based on the four optimal parameters calculated by the tactile optimal parameter adjustment module 1. This tactile expression acts in coordination with the background tactile vibration, which can effectively enhance the user's emotional experience. It should be noted that the background tactile vibration persists throughout the experiment, and it automatically adjusts the vibration intensity and rhythm according to the volume of the currently playing audio. Specifically, the vibration intensity is positively correlated with the audio volume, and at the same time, a certain volume threshold is set to adjust the vibration rhythm. When the audio volume is lower than this threshold, no vibration is generated. The purpose of the tactile background design is the real-time interaction between the tactile vibration and the audio content. By adjusting the vibration intensity and rhythm, an immersive emotional experience is brought to the user. On the other hand, the tactile vibration parameters determined by the tactile optimal parameter adjustment module 1 are not affected by the audio content, but dynamically optimize parameters such as the vibration intensity and frequency according to the user's real-time emotional state to achieve a more accurate emotional regulation effect.

[0050] The visual and auditory generation module 3 provides visual and auditory stimuli for the user, including movie clip materials of different emotional types, guiding the user into a specific emotional state. Specifically, the audio-visual materials include 16 movie clips of about 4 minutes, covering 4 emotions: happiness, sadness, fear, and calmness, that is, each emotion corresponds to 4 movie clips. Among them, the audio of the movie clips provides the basis for the change of the background tactile vibration for the tactile generation module 2. These audio-visual stimuli act in coordination with the tactile stimuli, further enhancing the user's emotional experience.

[0051] Referring to Figure 4 , Figure 4It is a schematic diagram of the multi - physiological signal acquisition module 4. The multi - physiological signal acquisition module 4 acquires various physiological signals of the user in real - time, including 64 - channel electroencephalogram (EEG) signals and electrocardiogram (ECG) signals. Among them, the EEG signals are acquired by the EEG signal acquisition module 9, and the EEG signal acquisition module 9 consists of a 64 - lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier. The ECG signals are acquired by the ECG signal acquisition module 10, and the ECG signal acquisition module is an ActiveTwo series high - lead ECG acquisition system of Biosemi company.

[0052] Refer to Figure 5 , Figure 5 It is a schematic diagram of the multi - sensory signal acquisition module 5. The multi - sensory signal acquisition module 5 includes an auditory signal acquisition module 11 and a tactile signal acquisition module 12. The auditory signal acquisition module 11 can acquire the audio signals in the visual - auditory generation module 3 in real - time, and the tactile signal acquisition module 12 can acquire the tactile vibration signals in the tactile generation module 2 in real - time.

[0053] Refer to Figure 6 , Figure 6It is a schematic diagram of the multi-modal fusion emotion recognition module 6. The multi-modal fusion emotion recognition module 6 can analyze, process and recognize the user's current emotional state according to the user's multi-physiological signals and multi-sensory signals of the induced materials, and then send this emotional state signal to the tactile optimal parameter adjustment module 1 for intelligent regulation of tactile parameters. The multi-modal fusion emotion recognition module 6 includes a signal preprocessing module 13, a feature extraction module 14, a feature fusion module 15 and an emotion decoding module 16. The signal preprocessing module 13 preprocesses the collected electroencephalogram and electrocardiogram signals, including downsampling, filtering, artifact removal, etc. The feature extraction module 14 extracts features from the multi-sensory signals, preprocessed electroencephalogram and electrocardiogram signals respectively. Among them, the feature extraction methods of electroencephalogram signals include power spectral density, differential entropy, asymmetric difference of differential entropy, asymmetric quotient of differential entropy, offline wavelet analysis, statistical features (mean, variance), etc. The feature extraction methods of electrocardiogram signals include heart rate and heart rate variability, etc. The feature fusion module 15 uses a feature fusion algorithm to fuse the multi-physiological signal features of the feature extraction module 14 with the audio features extracted from the audio signal and the vibration features extracted from the tactile vibration signal. Among them, the feature fusion algorithms include weighted average, principal component analysis, deep belief network, etc. The emotion decoding module 16 uses a classification algorithm to classify the multi-modal fusion features obtained by the feature fusion module 15 to obtain the user's current emotional state. Among them, the classification algorithms include support vector machine, logistic regression, naive Bayes and deep learning, etc. The superiority of the multi-modal fusion emotion recognition module 6 lies in that considering the instability of physiological signals, especially electroencephalogram signals, the fusion of multi-physiological signal features with audio and tactile modal features can effectively reduce the impact of physiological signal instability on the emotion recognition result, and thus improve the performance of emotion recognition.

[0054] Embodiment 2:

[0055] The present invention provides a technical solution: an emotion tactile regulation method based on multi-modal fusion. Refer to Figure 7 , Figure 7 which is a flowchart of the emotion tactile regulation method based on multi-modal fusion, and the specific implementation steps are as follows:

[0056] Step S1: At the beginning of the experiment, apply audio-visual-tactile fusion stimuli. By presenting audio-visual stimuli and tactile stimuli, guide the user into different emotional states. Refer to Figure 8 , Figure 8It is an experimental paradigm diagram of the emotion tactile regulation system based on multimodal fusion. Among them, the audiovisual stimuli consist of 16 movie clips of about 4 minutes each, covering 4 emotions: happiness, sadness, fear, and calmness, with 4 movie clips corresponding to each emotion. The tactile stimuli include a continuously present background vibration that adapts to the audio, and a tactile effect with vibration parameters determined by the tactile optimal parameter adjustment module. After each movie clip is played, the user conducts a 20s self-subjective evaluation, that is, the actual feeling of the movie clip, to verify the effectiveness of the experiment. After that, the user will rest for 30s to prepare for the next round of clip playback.

[0057] Step S2: Multimodal acquisition, including multi-physiological signals and multi-sensory signals. The electroencephalogram (EEG) and electrocardiogram (ECG) signals, audio signals, and tactile vibration signals of the user are acquired in real time. The EEG signals are acquired by an EEG acquisition module composed of a 64-channel wet electrode cap and an EEG acquisition amplifier of the Brain Products GmbH series. The ECG signals are acquired by an ECG signal acquisition module constituted by an ActiveTwo series high-lead ECG acquisition system; the audio signals are acquired by an auditory signal acquisition module, and the tactile vibration signals are acquired by a tactile signal acquisition module.

[0058] Step S3: Multimodal feature extraction and feature fusion. For the acquired EEG and ECG signals, preprocessing is first performed, including downsampling, filtering, artifact removal, etc., to ensure the quality and stability of the signals. Then, features are extracted from the preprocessed EEG and ECG signals to obtain the EEG signal features and ECG signal features of the user, which can capture the change patterns of different physiological signals in different emotional states. At the same time, audio features are extracted from the audio signals, and vibration features are extracted from the tactile vibration signals. After that, a feature fusion algorithm is used to fuse the multi-physiological signal features with the audio features and tactile vibration features to enhance the accuracy and robustness of emotion state recognition.

[0059] Step S4: Multimodal fusion emotion decoding and feedback. A classification algorithm is used to classify the multimodal fusion features to obtain the user's current emotional state, and it is fed back to the tactile optimal parameter adjustment module.

[0060] Step S5: Solve and update the tactile vibration parameters. The tactile optimal parameter adjustment module receives the user's current emotional state, automatically solves the optimal tactile parameters according to the difference from the target emotion, and sends them to the tactile generation module to generate a tactile effect, so as to ensure that the applied tactile stimulus matches the user's actual emotional needs.

[0061] Step S6: End of the experiment, establish an emotional touch database. After the playback of 16 movie clips ends, that is, after the entire experiment ends, analyze the vibration parameters corresponding to different emotional states of the user, and establish an emotional touch database for the user. Map different vibration parameters to different emotional states, and generate personalized tactile patterns to present diverse emotional experiences to the user.

[0062] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can still be made, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. An emotional touch regulation system based on multimodal fusion, characterized in that: It includes a tactile optimal parameter adjustment module, a tactile generation module, a visual and auditory generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module, and a multi-modal fusion emotion recognition module; the tactile optimal parameter adjustment module automatically solves the optimal tactile parameters according to the difference between the user's current emotion and the target emotion, sends them to the tactile generation module and generates a tactile effect, and the tactile generation module cooperates with the visual and auditory generation module to generate visual, auditory and tactile fusion stimuli acting on the user to regulate the user's emotion. The multi-physiological signal acquisition module and the multi-sensory signal acquisition module collect various physiological signals, audio signals and tactile vibration signals of the user in real time, input them into the multi-modal fusion emotion recognition module to detect the user's current emotional state, and feedback to the tactile optimal parameter adjustment module to form a closed-loop emotion regulation system; The tactile optimal parameter adjustment module, as the core of the emotion tactile regulation system based on multi-modal fusion, automatically finds tactile parameters according to the difference between the user's current emotional state and the target emotion with the help of optimization theory, and sends the parameters to the tactile generation module to ensure the effectiveness of emotion regulation; the tactile optimal parameter adjustment module includes a tactile parameter optimization model and a tactile parameter solving module; The tactile parameter optimization model is expressed as s.t. 0 ≤ P ≤ P m 0≤f≤f m 0≤q≤q m 0≤r≤r m Among them, M i is the actual power value of a certain electrode i on the electroencephalogram topographic map, M bi is the reference power value of a certain electrode i on the electroencephalogram topographic map, S is the calculated emotional state value, S b is the target emotional state value, P is the actual power value consumed by the tactile generation module, P m is the maximum power value set for the tactile generation module to consume, f m is the set maximum tactile vibration frequency, q m is the set maximum tactile vibration intensity, r m is the set maximum tactile vibration rhythm, γ, μ, and φ are the weight coefficients of the model.

2. The emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The tactile parameter solving module uses machine learning algorithms to solve the four parameters in the tactile optimal parameter model, namely the tactile vibration frequency f, the tactile vibration intensity q, the tactile vibration rhythm r, and the tactile vibration position c, and sends the solved parameters to the tactile generation module.

3. The emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The tactile generation module is a wearable device that expresses touch through vibration, including a vibrating vest, a vibrating bracelet, and a vibrating glove. By setting the vibration frequency, intensity, rhythm, and position of the tactile generation device, it conveys a specific tactile experience; the feature of the tactile generation module is that there is always background tactile vibration that changes adaptively with the audio, and at the same time, another tactile expression can be realized based on the four optimal parameters calculated by the tactile parameter optimization model. This tactile expression cooperates with the background tactile vibration to enhance the user's emotional experience.

4. The emotional touch regulation system based on multimodal fusion according to claim 3, characterized in that: The visual and auditory generation module provides visual and auditory stimuli for the user, including movie clip materials of different emotion types; these audio-visual stimulus materials help to guide the user into a specific emotional state, and the audio of the movie clip is used as the basis for the change of the background tactile vibration.

5. The emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The multi-physiological signal acquisition module collects various physiological signals of the user in real time, including 64-channel electroencephalogram (EEG) signals and electrocardiogram (ECG) signals; among them, the EEG signals are collected by the EEG signal acquisition module, and the EEG signal acquisition module consists of a 64-lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier; the ECG signals are collected by the ECG signal acquisition module, and the ECG signal acquisition module is the ActiveTwo series high-lead ECG acquisition system of Biosemi company.

6. The emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The multi-sensory signal acquisition module includes an auditory signal acquisition module and a tactile signal acquisition module, which can respectively and real-time acquire the audio signals in the visual-auditory generation module and the tactile vibration signals in the tactile generation module.

7. The emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The multi-modal fusion emotion recognition module can analyze, process and recognize the user's current emotional state according to the user's multi-physiological signals and the multi-sensory signals of the induction materials, and then send this emotional state signal to the tactile optimal parameter adjustment module for intelligent regulation of tactile parameters; The multi-modal fusion emotion recognition module includes a signal preprocessing module, a feature extraction module, a feature fusion module and an emotion decoding module; The signal preprocessing module preprocesses the collected EEG and ECG signals, including downsampling, filtering and artifact removal; the feature extraction module respectively extracts features from the auditory signals, tactile signals, preprocessed EEG and ECG signals; the feature fusion module uses a feature fusion algorithm to fuse the multi-physiological signal features with the audio features extracted from the audio signals and the vibration features extracted from the tactile vibration signals; The emotion decoding module classifies the multi-modal fusion features using a classification algorithm to obtain the user's current emotional state; the superiority of the multi-modal fusion emotion recognition module lies in the fusion of multi-physiological signal features with audio and tactile modality features, reducing the impact of physiological signal instability on the emotion recognition result.

8. The regulation method generated by the emotional touch regulation system based on multimodal fusion according to claim 1, characterized in that: The specific implementation steps are as follows: Step S1: Apply visual-auditory-tactile fusion stimuli By presenting visual-auditory stimuli and tactile stimuli, guide the user into different emotional states; The visual-auditory stimuli consist of 16 four-minute movie clips, covering 4 emotions: happy, sad, fearful and calm, with 4 movie clips corresponding to each emotion; The tactile stimuli include a continuously present background vibration that adapts to the audio, and a tactile effect with vibration parameters determined by the tactile optimal parameter adjustment module; After each movie clip is played, the user conducts a 20s self-subjective evaluation, that is, the actual feeling of this movie clip, to verify the effectiveness of the experiment; thereafter, the user will rest for 30s to prepare for the next round of clip playback; Step S2: Multi-modal acquisition, including multi-physiological signals and multi-sensory signals Real-time acquire the user's EEG and ECG signals, audio signals and tactile vibration signals; the EEG signals are acquired by an EEG signal acquisition module composed of a 64-channel wet electrode cap and a Brain Products GmbH series EEG acquisition amplifier; the ECG signals are acquired by an ECG signal acquisition module composed of an ActiveTwo series high-channel ECG acquisition system, the audio signals are acquired by the auditory signal acquisition module, and the tactile vibration signals are acquired by the tactile signal acquisition module; Step S3: Multi-modal feature extraction and feature fusion For the collected EEG and ECG signals, preprocessing is first performed, including downsampling, filtering, and artifact removal, to ensure the quality and stability of the signals; then, feature extraction is performed from the preprocessed EEG and ECG signals to obtain the EEG signal features and ECG signal features of the user, which can capture the change patterns of different physiological signals in different emotional states; at the same time, audio features are extracted from the audio signals, and vibration features are extracted from the tactile vibration signals; After that, a feature fusion algorithm is used to fuse the multi-physiological signal features with the audio features and tactile vibration features to enhance the accuracy and robustness of emotion state recognition; Step S4: Multi-modal fusion emotion decoding and feedback Use a classification algorithm to classify the multi-modal fusion features to obtain the user's current emotional state and feedback it to the tactile optimal parameter adjustment module; Step S5: Solve and update tactile vibration parameters The tactile optimal parameter adjustment module receives the user's current emotional state, automatically solves the optimal tactile parameters according to the difference from the target emotion, and sends them to the tactile generation module to generate a tactile effect to ensure that the applied tactile stimulus matches the user's actual emotional needs; Step S6: End of the experiment, establish an emotional touch database After the playback of 16 movie clips ends, that is, after the entire experiment ends, analyze the vibration parameters corresponding to different emotional states of the user, and establish an emotional touch database for the user; map different vibration parameters to different emotional states and generate personalized tactile patterns to present diverse emotional experiences to the user.

Images