Emotional haptic regulation system and method based on multi-modal fusion

Abstract

Disclosed in the present invention are an emotional haptic regulation system and method based on multi-modal fusion. The system comprises a haptic optimal parameter adjustment module, a haptic generation module, a visual-auditory generation module, a multi-physiological signal collection module, a multi-sensory signal collection module and a multi-modal fusion emotion recognition module. The system collects multiple physiological signals of a user, fuses features of the multiple physiological signals with audio and haptic modality features, accurately recognizes the current emotional state of the user in real time by using advanced data processing and analysis techniques, automatically searches for haptic parameters by means of an optimization theory, and actively regulates the emotional state by applying haptic stimuli to the user; and the limitations of conventional methods such as subjective scales are overcome, the impact of unstable physiological signals on emotion recognition results is effectively reduced, and the accuracy of emotion detection in the emotional haptic regulation system is significantly improved. The emotional haptic regulation system has broad application potential in the fields of personal emotion management, medical rehabilitation, audio-video entertainment, etc.

Description

Emotional tactile control system and method based on multimodal fusion Technical Field

[0001] The present invention belongs to the technical field of emotion regulation, and specifically relates to an emotion tactile regulation system and method based on multimodal fusion. Background Art

[0002] In recent years, the rapid development of affective computing and haptic technology has spawned a new field: affective haptics. Affective computing aims to reveal the mechanisms underlying the generation and expression of emotions, while haptic technology focuses on simulating the human tactile perception process. The field of affective haptics, which integrates emotional information with haptic technology, aims to explore the potential of using haptic technology in the detection, display, and communication of emotions, offering new possibilities for human-computer interaction.

[0003] As a core technology in the field of emotional tactile perception, the emotional tactile control system aims to perceive an individual's emotional state in real time and guide it through tactile stimulation, thereby enabling active emotional regulation during interaction. The emotional tactile control system has promising applications in fields of human-computer interaction, such as medical rehabilitation and audio-visual entertainment. For example, in healthcare, the system uses tactile stimulation to assist in emotional regulation, providing an innovative approach for the treatment of affective disorders such as depression. In audio-visual entertainment, the system enhances emotional immersion through tactile stimulation, creating a more immersive experience for users.

[0004] However, despite the broad prospects of the emotional tactile control system, there are still some technical challenges that need to be solved:

[0005] 1. Lack of objective and real-time emotion detection methods. Existing emotion detection methods in emotional tactile control systems often rely on subjective evaluation methods such as scales. This method is easily interfered with by individual subjective consciousness and external environment, resulting in limited objectivity and real-time performance of emotional states.

[0006] 2. The mechanism of influence between tactile stimulation and emotional state is still unclear. Although there is a close connection between touch and emotion, there is still a lack of clear guidance on how to adjust specific tactile parameters to achieve specific emotional regulation. As a result, the system often fails to achieve the expected emotional regulation effect in practical applications.

[0007] The above two technical difficulties restrict the further development of the emotional tactile regulation system.

[0008] Summary of the Invention

[0009] To solve the above problems, the present invention discloses an emotional tactile regulation system and method based on multimodal fusion. The system collects multiple physiological signals of the user, such as EEG and ECG, and fuses the characteristics of multiple physiological signals with audio and tactile modal characteristics. Combined with advanced data processing and analysis technologies, it can accurately identify the user's current emotional state in real time, and automatically search for tactile parameters with the help of optimization theory, and realize active regulation of the emotional state by applying tactile stimulation to the user.

[0010] To achieve the above object, the technical solution of the present invention is as follows:

[0011] The emotional tactile control system based on multimodal fusion includes a tactile optimal parameter adjustment module, a tactile generation module, an audiovisual generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module and a multimodal fusion emotion recognition module. The tactile optimal parameter adjustment module automatically solves the optimal tactile parameters based on the difference between the user's current emotion and the target emotion, sends them to the tactile generation module and produces a tactile effect. The tactile generation module and the audiovisual generation module work together to produce audiovisual and tactile fusion stimulation to act on the user to regulate the user's emotions. The multi-physiological signal acquisition module and the multi-sensory signal acquisition module collect the user's various physiological signals, audio signals and tactile vibration signals in real time, input them into the multimodal fusion emotion recognition module to detect the user's current emotional state, and feed them back to the tactile optimal parameter adjustment module to form a closed-loop emotion control system.

[0012] The tactile optimal parameter adjustment module is the core of the emotional tactile control system based on multimodal fusion. It automatically finds tactile parameters based on the difference between the user's current emotional state and the target emotion with the help of optimization theory, and sends the parameters to the tactile generation module to ensure the effectiveness of emotional control. The tactile optimal parameter adjustment module includes a tactile parameter optimization model and a tactile parameter solution module. The tactile parameter optimization model is expressed as

[0013] Among them, M i is the actual power value of a certain electrode i on the brain topography, M bi is the reference power value of a certain electrode i on the brain topography, S is the calculated emotional state value, S b is the target emotional state value, P is the actual power value consumed by the tactile generation module, and P m It is to set the maximum power consumption of the tactile module, f m is to set the maximum tactile vibration frequency, q m is to set the maximum tactile vibration intensity, r m is to set the maximum tactile vibration rhythm, γ, μ and φ are the weight coefficients of the model.

[0014] The tactile parameter solving module uses machine learning algorithms such as particle swarm optimization algorithm and reinforcement learning to solve the four parameters in the tactile optimal parameter model, namely tactile vibration frequency f, tactile vibration intensity q, tactile vibration rhythm r and tactile vibration position c, and sends the solved parameters to the tactile generation module.

[0015] The tactile generation module is a type of wearable device that uses vibration to express tactile sensations, including vibrating vests, vibrating bracelets, and vibrating gloves. By setting the vibration frequency, intensity, rhythm, and position of the tactile generation device, a specific tactile experience can be conveyed. The tactile generation module is characterized by a constant presence of background tactile vibrations that adaptively change with the audio. Furthermore, it can achieve a different tactile expression based on the four optimal parameters calculated by the tactile parameter optimization model. This tactile expression, in synergy with the background tactile vibrations, can effectively enhance the user's emotional experience.

[0016] The audiovisual generation module provides visual and auditory stimulation to the user, including film clips of different emotional types. These audiovisual stimulation materials help guide the user into a specific emotional state, wherein the audio of the film clip serves as the basis for the change of the background tactile vibration.

[0017] The multi-physiological signal acquisition module collects multiple physiological signals from the user in real time, including 64 channels of EEG and ECG signals. EEG signals are collected by the EEG signal acquisition module, which consists of a 64-lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier. ECG signals are collected by the ECG signal acquisition module, which uses Biosemi's ActiveTwo series high-lead ECG acquisition system.

[0018] The multi-sensory signal acquisition module includes an auditory signal acquisition module and a tactile signal acquisition module, which can respectively acquire the audio signal in the audio-visual generation module and the tactile vibration signal in the tactile generation module in real time.

[0019] The multimodal fusion emotion recognition module analyzes, processes, and identifies the user's current emotional state based on the user's multiple physiological signals and the multisensory signals of the induced material. This emotional state signal is then sent to the tactile optimal parameter adjustment module for intelligent control of tactile parameters. The multimodal fusion emotion recognition module comprises a signal preprocessing module, a feature extraction module, a feature fusion module, and an emotion decoding module. The signal preprocessing module performs preprocessing on the collected EEG and ECG signals, including downsampling, filtering, and artifact removal. The feature extraction module extracts features from the audio signal, tactile vibration signal, and preprocessed EEG and ECG signals. The feature fusion module uses a feature fusion algorithm to fuse the multimodal signal features with the audio features extracted from the audio signal and the vibration features extracted from the tactile vibration signal. The emotion decoding module uses a classification algorithm to classify the multimodal fusion features to determine the user's current emotional state. The advantage of the multimodal fusion emotion recognition module lies in its integration of multimodal signal features with audio and tactile modal features, effectively mitigating the impact of physiological signal instability on emotion recognition results.

[0020] The beneficial effects of the present invention include:

[0021] 1. The emotional tactile regulation system of the present invention adopts an emotion detection method based on multimodal fusion, which integrates multiple physiological signal features with audio and tactile modal features to achieve objective and real-time emotion recognition, overcomes the limitations of traditional subjective scales and other methods, effectively reduces the impact of physiological signal instability on emotion recognition results, and significantly improves the accuracy of emotion detection in the emotional tactile regulation system.

[0022] 2. The tactile optimal parameter adjustment module in the present invention analyzes the difference between the real-time emotional state and the target emotion, and automatically searches for appropriate tactile parameters with the help of optimization theory, so that tactile stimulation can more accurately guide and adjust the user's emotional state, effectively improving the effectiveness and efficiency of the emotional tactile regulation system.

[0023] 3. The emotional tactile control system based on multimodal fusion in the present invention can establish an emotional tactile database of users, and use big data and large model learning technology to generate personalized tactile patterns to present customized emotional experiences. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a block diagram of the principle of the emotional tactile control system based on multimodal fusion of the present invention;

[0025] FIG2 is a schematic diagram of a tactile optimal parameter adjustment module according to the present invention;

[0026] FIG3 is a schematic diagram of a tactile generating module of the present invention;

[0027] FIG4 is a schematic diagram of a multi-physiological signal acquisition module of the present invention;

[0028] FIG5 is a schematic diagram of a multi-sensory signal acquisition module of the present invention;

[0029] FIG6 is a schematic diagram of a multimodal fusion emotion recognition module of the present invention;

[0030] FIG7 is a flow chart of the emotional tactile control method based on multimodal fusion of the present invention;

[0031] FIG8 is a diagram showing the experimental paradigm of the tactile emotion regulation system of the present invention.

[0032] List of accompanying figures: 1. Tactile optimal parameter adjustment module; 2. Tactile generation module; 3. Visual and auditory generation module; 4. Multi-physiological signal acquisition module; 5. Multi-sensory signal acquisition module; 6. Multimodal fusion emotion recognition module; 7. Tactile parameter optimization model; 8. Tactile parameter solution module; 9. EEG signal acquisition module; 10. ECG signal acquisition module; 11. Auditory signal acquisition module; 12. Tactile signal acquisition module; 13. Signal preprocessing module; 14. Feature extraction module; 15. Feature fusion module; 16. Emotion decoding module. DETAILED DESCRIPTION

[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0034] Example 1:

[0035] This example describes an emotional tactile control system based on multimodal fusion, the overall principle diagram of which is shown in Figure 1. It includes a tactile optimal parameter adjustment module 1, a tactile generation module 2, an audiovisual generation module 3, a multi-physiological signal acquisition module 4, a multi-sensory signal acquisition module 5, and a multimodal fusion emotion recognition module 6. The tactile optimal parameter adjustment module 1 automatically solves the optimal tactile parameters based on the difference between the user's current emotion and the target emotion, sends them to the tactile generation module 2 and produces a tactile effect. The tactile generation module 2 and the audiovisual generation module 3 work together to produce audiovisual tactile fusion stimulation to act on the user to regulate the user's emotion. The multi-physiological signal acquisition module 4 and the multi-sensory signal acquisition module 5 collect the user's various physiological signals, audio signals, and tactile vibration signals in real time, input them into the multimodal fusion emotion recognition module 6 to detect the user's current emotional state, and feed them back to the tactile optimal parameter adjustment module 1, forming a closed-loop emotional tactile control system based on multimodal fusion.

[0036] Referring to Figure 2, Figure 2 is a schematic diagram of the tactile optimal parameter adjustment module 1. As the core of the multimodal fusion-based emotional tactile regulation system, the tactile optimal parameter adjustment module 1 automatically seeks tactile parameters based on the difference between the user's current emotional state and the target emotion using optimization theory, and sends the parameters to the tactile generation module 2 to ensure the effectiveness of emotional regulation. The tactile optimal parameter adjustment module 1 includes a tactile parameter optimization model 7 and a tactile parameter solution module 8.

[0037] The tactile parameter optimization model 7 integrates factors such as EEG signals, emotional state, and vibration power consumption to establish a reliable mathematical model for adaptively adjusting four key parameters in the tactile generation module 2, including tactile vibration frequency f, tactile vibration intensity q, tactile vibration rhythm r, and tactile vibration position c. The tactile parameter optimization model 7 is expressed as

[0038] Among them, M i is the actual power value of a certain electrode i on the brain topography, M bi is the reference power value of a certain electrode i on the brain topography, S is the calculated emotional state value, S b is the target emotional state value, P is the actual power value consumed by the tactile generation module, and P m It is to set the maximum power consumption of the tactile module, f m is to set the maximum tactile vibration frequency, q m is to set the maximum tactile vibration intensity, r m is to set the maximum tactile vibration rhythm, γ, μ and is the weight coefficient of the model.

[0039] The tactile parameter solving module 8 includes optimization algorithms, machine learning, reinforcement learning and other methods, which are used to solve the four parameters in the tactile parameter optimization model 7, namely, tactile vibration frequency f, tactile vibration intensity q, tactile vibration rhythm r and tactile vibration position c, and send the solved parameters to the tactile generation module 2.

[0040] Refer to Figure 3, which is a schematic diagram of the tactile generating module 2. The tactile generating module 2 is a wearable device that uses vibration to express tactile sensations, including vibrating vests, vibrating bracelets, and vibrating gloves. By setting the vibration frequency, intensity, rhythm, and position of the tactile generating device, a specific tactile experience is delivered to the user. A characteristic of the tactile generating module 2 is that background tactile vibrations are always present and adaptively change with the audio. Furthermore, another tactile expression can be achieved based on the four optimal parameters calculated by the tactile optimal parameter adjustment module 1. This tactile expression synergizes with the background tactile vibrations to effectively enhance the user's emotional experience. It should be noted that the background tactile vibrations persist throughout the experiment, automatically adjusting their intensity and rhythm based on the volume of the audio being played. Specifically, the vibration intensity is positively correlated with the audio volume, and the vibration rhythm is adjusted by setting a certain volume threshold; when the audio volume falls below this threshold, no vibration occurs. The purpose of the tactile background design is to ensure real-time interaction between the tactile vibrations and the audio content, providing the user with an immersive emotional experience through the adjustment of the vibration intensity and rhythm. On the other hand, the tactile vibration parameters determined by the tactile optimal parameter adjustment module 1 are not affected by the audio content, but dynamically optimize parameters such as vibration intensity and frequency according to the user's real-time emotional state to achieve a more accurate emotional regulation effect.

[0041] The audiovisual generation module 3 provides users with visual and auditory stimulation, including film clips of different emotional types, to guide users into specific emotional states. Specifically, the audiovisual materials include 16 four-minute movie clips covering the four emotions of happiness, sadness, fear, and calmness, with four movie clips corresponding to each emotion. The audio of the movie clips provides the basis for changes in background tactile vibrations in the tactile generation module 2. These audiovisual stimuli work synergistically with the tactile stimulation to further enhance the user's emotional experience.

[0042] Referring to Figure 4, a schematic diagram of the multi-physiological signal acquisition module 4 is shown. The multi-physiological signal acquisition module 4 collects multiple physiological signals from the user in real time, including 64 channels of EEG and ECG signals. EEG signals are collected by an EEG signal acquisition module 9, which consists of a 64-lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier. ECG signals are collected by an ECG signal acquisition module 10, which is a Biosemi ActiveTwo series high-lead ECG acquisition system.

[0043] Referring to Figure 5, Figure 5 is a schematic diagram of the multi-sensory signal acquisition module 5. The multi-sensory signal acquisition module 5 includes an auditory signal acquisition module 11 and a tactile signal acquisition module 12. The auditory signal acquisition module 11 can acquire audio signals from the audiovisual generation module 3 in real time, and the tactile signal acquisition module 12 can acquire tactile vibration signals from the tactile generation module 2 in real time.

[0044] Referring to Figure 6, Figure 6 is a schematic diagram of the multimodal fusion emotion recognition module 6. The multimodal fusion emotion recognition module 6 can analyze, process, and identify the user's current emotional state based on the user's multi-physiological signals and the multi-sensory signals of the induced material, and then send this emotional state signal to the tactile optimal parameter adjustment module 1 for intelligent regulation of tactile parameters. The multimodal fusion emotion recognition module 6 includes a signal preprocessing module 13, a feature extraction module 14, a feature fusion module 15, and an emotion decoding module 16. The signal preprocessing module 13 preprocesses the collected EEG and ECG signals, including downsampling, filtering, and artifact removal. The feature extraction module 14 performs feature extraction on the multisensory signals and the preprocessed EEG and ECG signals respectively. Among them, the feature extraction methods of EEG signals include power spectral density, differential entropy, asymmetric difference of differential entropy, asymmetric quotient of differential entropy, offline wavelet analysis, statistical features (mean, variance), etc. The feature extraction methods of ECG signals include heart rate and heart rate variability, etc. The feature fusion module 15 utilizes a feature fusion algorithm to fuse the multi-physiological signal features of the feature extraction module 14 with the audio features extracted from the audio signal and the vibration features extracted from the tactile vibration signal. Among them, the feature fusion algorithm includes weighted average, principal component analysis, deep belief network, etc. The emotion decoding module 16 utilizes a classification algorithm to classify the multi-modal fusion features obtained by the feature fusion module 15 to obtain the current emotional state of the user. Among them, the classification algorithm includes support vector machine, logistic regression, naive Bayes and deep learning, etc. The advantage of the multi-modal fusion emotion recognition module 6 is that it takes into account the instability of physiological signals, especially EEG signals, and fuses the multi-physiological signal features with audio and tactile modal features, which can effectively reduce the impact of physiological signal instability on emotion recognition results, thereby improving the performance of emotion recognition.

[0045] Example 2:

[0046] The present invention provides a technical solution: an emotional tactile control method based on multimodal fusion. Referring to Figure 7, Figure 7 is a flow chart of the emotional tactile control method based on multimodal fusion. The specific implementation steps are as follows:

[0047] Step S1: The experiment starts, and audiovisual tactile fusion stimulation is applied. By presenting audiovisual stimulation and tactile stimulation, the user is guided into different emotional states. Referring to Figure 8, Figure 8 is an experimental paradigm diagram of the emotional tactile regulation system based on multimodal fusion. Among them, the audiovisual stimulation consists of 16 movie clips of about 4 minutes, covering 4 emotions of happiness, sadness, fear and calmness, and each emotion corresponds to 4 movie clips. The tactile stimulation includes persistent background vibrations that change adaptively with the audio, and the tactile effect of the vibration parameters determined by the tactile optimal parameter adjustment module. After each movie clip is played, the user conducts a 20-second self-subjective evaluation, that is, the actual feeling of the movie clip, which is used to verify the validity of the experiment. After that, the user will rest for 30 seconds to prepare for the next round of clip playback.

[0048] Step S2: Multimodal acquisition, including multiple physiological and multisensory signals. Real-time acquisition of the user's EEG and ECG signals, audio signals, and tactile vibration signals. EEG signals are collected by an EEG acquisition module consisting of a 64-lead wet electrode cap and a Brain Products GmbH series EEG acquisition amplifier. ECG signals are collected by an ECG signal acquisition module consisting of an ActiveTwo series high-lead ECG acquisition system. Audio signals are collected by an auditory signal acquisition module, and tactile vibration signals are collected by a tactile signal acquisition module.

[0049] Step S3: Multimodal feature extraction and feature fusion. The collected EEG and ECG signals are first preprocessed, including downsampling, filtering, and artifact removal, to ensure the quality and stability of the signals. Feature extraction is then performed from the preprocessed EEG and ECG signals to obtain the user's EEG signal features and ECG signal features. These features can capture the changing patterns of different physiological signals under different emotional states. At the same time, audio features are extracted from audio signals, and tactile vibration features are extracted from tactile vibration signals. Afterwards, the feature fusion algorithm is used to fuse the multi-physiological signal features with the audio features and tactile vibration features to enhance the accuracy and robustness of emotional state recognition.

[0050] Step S4: Multimodal fusion emotion decoding and feedback: The multimodal fusion features are classified using a classification algorithm to obtain the user's current emotional state and feedback to the tactile optimal parameter adjustment module.

[0051] Step S5: Calculate and update tactile vibration parameters. The tactile optimal parameter adjustment module receives the user's current emotional state and automatically calculates the optimal tactile parameters based on the difference from the target emotion. The optimal tactile parameters are sent to the tactile generation module to generate tactile effects, ensuring that the applied tactile stimulation matches the user's actual emotional needs.

[0052] Step S6: The experiment concludes and an emotional tactile database is established. After the 16 movie clips have finished playing, the entire experiment is concluded. The vibration parameters corresponding to the user's different emotional states are analyzed to establish the user's emotional tactile database. Different vibration parameters are mapped to different emotional states, and personalized tactile patterns are generated, presenting the user with a diverse emotional experience.

[0053] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. An emotional tactile control system based on multimodal fusion, characterized by: It includes a tactile optimal parameter adjustment module, a tactile generation module, an audio-visual generation module, a multi-physiological signal acquisition module, a multi-sensory signal acquisition module and a multi-modal fusion emotion recognition module. The tactile optimal parameter adjustment module automatically solves the optimal tactile parameters according to the difference between the user's current emotion and the target emotion, sends them to the tactile generation module and produces a tactile effect. The tactile generation module and the audio-visual generation module cooperate to produce audio-visual tactile fusion stimulation to act on the user to regulate the user's emotion. The multi-physiological signal acquisition module and the multi-sensory signal acquisition module collect the user's various physiological signals, audio signals and tactile vibration signals in real time, input them into the multi-modal fusion emotion recognition module to detect the user's current emotional state, and feed them back to the tactile optimal parameter adjustment module to form a closed-loop emotion regulation system.

2. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The tactile optimal parameter adjustment module serves as the core of the emotional tactile regulation system based on multimodal fusion. According to the difference between the user's current emotional state and the target emotion, it automatically searches for tactile parameters with the help of optimization theory, and sends the parameters to the tactile generation module to ensure the effectiveness of emotional regulation. The tactile optimal parameter adjustment module includes a tactile parameter optimization model and a tactile parameter solving module.

3. The emotional tactile control system based on multimodal fusion according to claim 2, characterized in that: The tactile parameter optimization model is expressed as st0≤P≤P m 0≤f≤f m 0≤q≤q m 0≤r≤r m Among them, M i is the actual power value of a certain electrode i on the brain topography, M bi is the reference power value of a certain electrode i on the brain topography, S is the calculated emotional state value, S b is the target emotional state value, P is the actual power value consumed by the tactile generation module, and P m It is the maximum power value consumed by the tactile module. m is to set the maximum tactile vibration frequency, q m is to set the maximum tactile vibration intensity, r m is to set the maximum tactile vibration rhythm, γ, μ and φ are the weight coefficients of the model.

4. The emotional tactile control system based on multimodal fusion according to claim 2, characterized in that: The tactile parameter solving module adopts a machine learning algorithm to solve four parameters in the tactile optimal parameter model, namely, tactile vibration frequency f, tactile vibration intensity q, tactile vibration rhythm r and tactile vibration position c, and sends the solved parameters to the tactile generation module.

5. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The tactile generating module is a type of wearable device that expresses touch through vibration, including a vibrating vest, a vibrating bracelet, and a vibrating glove. By setting the vibration frequency, intensity, rhythm, and position of the tactile generating device, a specific tactile experience is conveyed; the tactile generating module is characterized in that there is always a background tactile vibration that changes adaptively with the audio, and at the same time, another tactile expression can be achieved based on the four optimal parameters calculated by the tactile parameter optimization model. The tactile expression works synergistically with the background tactile vibration to enhance the user's emotional experience.

6. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The audiovisual generation module provides visual and auditory stimulation to the user, including movie clip materials of different emotional types; these audiovisual stimulation materials help guide the user into a specific emotional state, wherein the audio of the movie clip serves as the basis for the change of the background tactile vibration.

7. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The multi-physiological signal acquisition module acquires multiple physiological signals of the user in real time, including 64-channel EEG signals and ECG signals; wherein, the EEG signals are acquired by the EEG signal acquisition module, and the EEG signal acquisition module is composed of a 64-lead actiCAP electrode cap and a Brain Products GmbH series EEG amplifier; the ECG signals are acquired by the ECG signal acquisition module, and the ECG signal acquisition module is Biosemi's ActiveTwo series high-lead ECG acquisition system.

8. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The multi-sensory signal acquisition module includes an auditory signal acquisition module and a tactile signal acquisition module, which can respectively acquire the audio signal in the audio-visual generation module and the tactile vibration signal in the tactile generation module in real time.

9. The emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The multimodal fusion emotion recognition module can analyze, process and identify the user's current emotional state based on the user's multi-physiological signals and the multi-sensory signals of the induced material, and then send the emotional state signal to the tactile optimal parameter adjustment module to perform intelligent regulation of the tactile parameters; The multimodal fusion emotion recognition module includes a signal preprocessing module, a feature extraction module, a feature fusion module and an emotion decoding module; The signal preprocessing module preprocesses the collected EEG and ECG signals, including downsampling, filtering, and removing artifacts; the feature extraction module extracts features from the audio signal, the tactile vibration signal, and the preprocessed EEG and ECG signals respectively; the feature fusion module uses a feature fusion algorithm to fuse the multi-physiological signal features with the audio features extracted from the audio signal and the vibration features extracted from the tactile vibration signal; The emotion decoding module uses a classification algorithm to classify the multimodal fusion features to obtain the user's current emotional state; the advantage of the multimodal fusion emotion recognition module lies in the fusion of multiple physiological signal features with audio and tactile modality features, reducing the impact of physiological signal instability on emotion recognition results.

10. The control method generated by the emotional tactile control system based on multimodal fusion according to claim 1, characterized in that: The specific implementation steps are as follows: Step S1: Applying audio-visual fusion stimulation By presenting audio-visual and tactile stimulation, users are guided into different emotional states; The audio-visual stimuli consisted of 16 4-minute movie clips covering four emotions: happiness, sadness, fear, and calmness. The emotions correspond to 4 movie clips; The tactile stimulation includes a continuous background vibration that changes adaptively with the audio, and a tactile effect whose vibration parameters are determined by a tactile optimal parameter adjustment module; After each movie clip is played, the user will conduct a 20-second self-assessment, that is, their actual feelings about the movie clip, to verify the effectiveness of the experiment; after that, the user will rest for 30 seconds to prepare for the next round of clip playback; Step S2: Multimodal acquisition, including multi-physiological signals and multi-sensory signals Collect the user's EEG and ECG signals, audio signals and tactile vibration signals in real time; EEG signals are collected by an EEG signal acquisition module consisting of a 64-lead wet electrode cap and a Brain Products GmbH series EEG acquisition amplifier; ECG signals are collected by an ECG signal acquisition module consisting of an ActiveTwo series high-lead ECG acquisition system, audio signals are collected by an auditory signal acquisition module, and tactile vibration signals are collected by a tactile signal acquisition module; Step S3: Multimodal feature extraction and feature fusion For the collected EEG and ECG signals, preprocessing is first performed, including downsampling, filtering, and artifact removal to ensure the quality and stability of the signals. Then, feature extraction is performed from the preprocessed EEG and ECG signals to obtain the user's EEG and ECG signal features, which can capture the changing patterns of different physiological signals under different emotional states. At the same time, audio features are extracted from audio signals, and tactile vibration features are extracted from tactile vibration signals. Afterwards, the feature fusion algorithm is used to fuse the multi-physiological signal features with the audio features and tactile vibration features to enhance the accuracy and robustness of emotion state recognition; Step S4: Multimodal fusion emotion decoding and feedback The multimodal fusion features are classified using a classification algorithm to obtain the user's current emotional state and feed it back to the tactile optimal parameter adjustment module; Step S5: Solving and updating tactile vibration parameters The tactile optimal parameter adjustment module receives the user's current emotional state, automatically solves the optimal tactile parameters according to the difference with the target emotion, sends them to the tactile generation module and generates a tactile effect to ensure that the applied tactile stimulation matches the user's actual emotional needs; Step S6: End of the experiment, establish the emotional tactile database After the 16 movie clips are played, that is, the entire experiment is over, the vibration parameters corresponding to the user's different emotional states are analyzed to establish the user's emotional tactile database; different vibration parameters are mapped to different emotional states, and personalized tactile patterns are generated to present users with diverse emotional experiences.

Images