Emotion recognition method based on multi-modal fusion neural network

Abstract

The invention relates to the technical field of virtual reality (VR) interaction and artificial intelligence, and provides an emotion recognition method based on a multi-modal fusion neural network, which comprises the following steps: S1, acquiring multi-modal physiological signal data including electroencephalogram signals, eye movement data, heart rate variability signals and skin electric response signals; s2, performing filtering, slicing and normalization processing on the obtained original signal through an edge calculation preprocessing unit to generate multi-channel time sequence data; s3, inputting the multi-channel time sequence data into a multi-modal fusion neural network model which comprises a spatial feature extraction layer, a time sequence feature extraction layer, a cross-modal attention fusion layer and a classification output layer and is used for extracting deep features of a multi-modal signal and identifying an emotional state of a user; the emotional state of the user in the virtual reality environment can be accurately recognized in real time, the virtual scene is dynamically adjusted according to the emotional state, and the reality sense and the participation degree of user experience are improved.

Description

Technical Field

[0001] This invention relates to the fields of virtual reality (VR) interaction and artificial intelligence technology, and in particular to an emotion recognition method based on a multimodal fusion neural network. Background Technology

[0002] With the widespread application of virtual reality technology, human-computer interaction interfaces are gradually evolving towards emotional and personalized approaches to achieve a deeper level of immersive experience. Current mainstream virtual reality interaction methods primarily rely on physical input mechanisms such as controller operation and head motion tracking. These methods can only capture the user's external behavioral intentions and cannot perceive changes in their internal emotional state in real time. This results in a lack of emotional adaptability in the virtual environment's response, significantly weakening the realism and engagement of the user experience. Existing emotion recognition technologies attempt to address this problem through a single modal approach, such as using electroencephalogram (EEG) signals to monitor neural activity or using cameras to analyze facial expression features. However, in typical use cases of virtual reality headsets, the physical structure of the device completely covers the user's facial area, rendering vision-based facial expression recognition technology completely ineffective. Furthermore, methods relying on a single physiological signal source (such as heart rate variability or skin conductance levels) are highly susceptible to motion artifacts when the user makes natural head movements or body movements. These artifacts include electrode contact noise caused by muscle contraction or signal drift caused by head shaking, resulting in large fluctuations and insufficient reliability in emotion recognition results. Furthermore, existing technical solutions generally lack the ability to deeply integrate multimodal data and fail to effectively combine the dynamic evolution characteristics in the time dimension with the correlation of multi-source physiological signals in the spatial dimension. Therefore, under the highly dynamic changes of the virtual reality environment, it is difficult to maintain the real-time performance and robustness of emotion classification, which seriously restricts the practical application of emotional virtual reality applications. Summary of the Invention

[0003] To overcome the above problems, the purpose of this invention is to provide an emotion recognition method based on a multimodal fusion neural network that can identify the user's emotional state in a virtual reality environment in real time and accurately, and dynamically adjust the virtual scene accordingly to enhance the realism and engagement of the user experience.

[0004] This invention employs the following approach: an emotion recognition method based on a multimodal fusion neural network, comprising: Step S1: Acquire multimodal physiological signal data, including electroencephalogram (EEG) signals, eye movement data, heart rate variability signals, and skin conductance response signals; Step S2: The acquired raw signal is filtered, sliced, and normalized by the edge computing preprocessing unit to generate multi-channel time series data; Step S3: Input the multi-channel time-series data into a multimodal fusion neural network model. The multimodal fusion neural network model includes a spatial feature extraction layer, a temporal feature extraction layer, a cross-modal attention fusion layer, and a classification output layer, which are used to extract deep features of multimodal signals and identify user emotional states. The spatial feature extraction layer uses a graph convolutional network or a convolutional neural network to encode the independent spatial features of each modality; the temporal feature extraction layer uses a long short-term memory network or a gated recurrent unit to model the temporal dependence; and the cross-modal attention fusion layer uses a multi-head attention mechanism to weightedly fuse features from different modalities to generate a cross-modal joint representation. The classification output layer outputs the probability distribution of emotion categories based on the cross-modal joint representation, which serves as the emotion recognition result.

[0005] Furthermore, it includes: a VR wearable device body, which includes a head-mounted display shell, a flexible pad that fits the face, and a headband; the multimodal physiological signal data, integrated on the VR wearable device body, used to simultaneously collect the user's electroencephalogram (EEG) signals, eye movement data, heart rate variability signals, and skin conductance response signals; and a VR content adaptive adjustment module, used to dynamically adjust the presentation parameters of the virtual reality scene according to the identified emotional state.

[0006] Furthermore, the multimodal physiological signal data specifically includes: At least four dry electrode EEG acquisition units are deployed at the corresponding Fp1 and Fp2 locations in the prefrontal lobe and T3 and T4 locations in the temporal lobe, respectively, for acquiring EEG signals; The binocular optical eye-tracking unit, consisting of an infrared light source and a high-speed camera, is deployed inside the head-mounted display at the position below the eye socket to collect changes in pupil diameter, blink frequency, and gaze trajectory. The photoelectric volumetric sensor consists of two dry electrodes deployed at the corresponding temple position to collect photoplethysmography (PPG) signals. The skin conductance sensor, consisting of two gold-plated dry electrodes, is deployed on one side of the cheek corresponding to the flexible pad to collect skin conductance levels.

[0007] Furthermore, the edge computing preprocessing unit is embedded inside the VR headset, specifically including: Analog-to-digital converters are used to convert analog physiological signals into 24-bit high-precision digital signals; The digital signal processor is equipped with a bandpass filter bank, a motion artifact adaptive filter, and a sliding window slicer to generate time-aligned multi-channel time-series samples.

[0008] Furthermore, the multimodal fusion neural network model consists of the following structure: The spatial feature extraction layer consists of four parallel, structurally independent one-dimensional convolutional neural network branches, corresponding to four modalities: electroencephalography (EEG), eye movement, pulse wave, and skin conductance. Each branch includes at least three convolutional blocks, and each convolutional block is composed of a stack of one-dimensional convolutional layers, batch normalization layers, ReLU activation layers, and max pooling layers. The temporal feature extraction layer is composed of a bidirectional long short-term memory network layer. Its input is the four modal feature sequences output by the spatial feature extraction module, and its output is a temporal feature vector with forward and backward context information. The bidirectional long short-term memory network layer contains 128 hidden units. Its input dimension is the sum of the dimensions of the feature vectors output by each modal branch, and its output dimension is 256. The cross-modal attention fusion layer includes a multi-head attention mechanism layer and a feature concatenation layer. The multi-head attention mechanism layer uses the temporal feature vector as the query, key, and value input to calculate the attention weight distribution of each modality. The feature concatenation layer concatenates the weighted features of each modality to form a fused feature vector. The multi-head attention mechanism layer uses four attention heads, each with a dimension of 64, and calculates the weight coefficients of each modality feature relative to the emotion recognition task by scaling dot product attention. The weight coefficients are updated in real time during the model inference stage. The classification output layer includes at least two fully connected layers and one Softmax output layer, which is used to map the fused feature vector to a probability distribution on a preset emotion category. The preset emotion category includes four quadrant categories: high arousal-high pleasure, high arousal-low pleasure, low arousal-high pleasure, and low arousal-low pleasure, as well as a neutral emotion category, for a total of five output categories.

[0009] Furthermore, the emotion state analysis unit is used to receive the probability distribution and output discrete emotion labels and continuous pleasure-arousal coordinate values; The real-time rendering engine interface is used to dynamically adjust the tone mapping, environmental sound effect parameters, narrative branch weights, or interaction difficulty coefficients of the virtual scene based on the emotion tags or coordinate values.

[0010] Furthermore, the motion artifact adaptive filter is configured with a real-time motion compensation algorithm based on eye-tracking signals. The algorithm uses acceleration data collected by the eye-tracking unit as a reference signal and filters out motion interference components in EEG signals and pulse wave signals through an adaptive least mean square algorithm.

[0011] Furthermore, the sliding window slicer is configured with a window length of 3 seconds, a sliding step size of 0.5 seconds, and a 75% overlap rate between adjacent windows. The slicer also performs timestamp alignment on each modal signal when generating samples to ensure the temporal consistency of multimodal data.

[0012] Furthermore, in the four parallel one-dimensional convolutional neural network branches: The convolution kernel size of the EEG signal branch is 3×1, the stride is 1, and the number of output channels is 32, 64, and 128 respectively. The convolution kernel size of the eye-tracking signal branch is 5×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The convolution kernel size of the pulse wave signal branch is 4×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The kernel size of the electrodermal signal branch is 8×1, the stride is 1, and the number of output channels is 8, 16, and 32 respectively. Each branch's pooling layer uses max pooling with a size of 2×1.

[0013] Furthermore, the left and right ends of the head-mounted display shell are respectively connected to a first connecting strap and a second connecting strap. The ends of the first connecting strap and the second connecting strap are connected via a connecting box. The connecting box has a communication port at both the front and rear ends. A drive motor is embedded in the left and right ends of the communication port. The output end of the drive motor is connected to a drive gear. The outer ends of the first connecting strap and the second connecting strap are provided with racks that mesh with the drive gears. The ends of the first connecting strap and the second connecting strap are provided with anti-detachment blocks. An auxiliary fixing strap is provided between the middle of the head-mounted display shell and the connecting box. The auxiliary fixing strap is provided with a telescopic adjustment buckle. Multiple connecting wires are connected to the head-mounted display shell. The ends of the connecting wires are connected to electrode plates.

[0014] The beneficial effects of this invention are as follows: By integrating multimodal physiological signal acquisition, edge computing preprocessing, and a multimodal fusion neural network, this invention achieves efficient and reliable emotion recognition. It can accurately identify the user's emotional state in a virtual reality environment in real time and dynamically adjust the virtual scene accordingly, enhancing the realism and engagement of the user experience. It achieves high precision and robustness: by fusing multimodal signals from EEG (central nervous system), eye movement (behavioral), ECG, and skin conductance (peripheral autonomic nervous system), it overcomes the vulnerability of single-modality signals to interference in VR environments. Furthermore, by dynamically optimizing modal weights using attention mechanisms, it significantly improves emotion recognition. Accuracy (experimentally verified to be over 92.7%); Low latency and immersive experience: Utilizing edge computing for front-end preprocessing, combined with a lightweight hybrid neural network model, end-to-end latency is less than 50ms, meeting the real-time interaction requirements of VR; Strong anti-interference capability: To address interference from user head movements in the VR environment, this invention incorporates motion compensation algorithms from eye-tracking data during the preprocessing stage to adaptively filter motion artifacts in EEG and ECG signals; Privacy protection: Key physiological features are initially extracted locally at the edge, with only anonymized feature vectors uploaded to the cloud or relying solely on local computing power, reducing the risk of biometric privacy leaks. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention.

[0016] Figure 2 This is a block diagram illustrating the principle of a multimodal fusion neural network model.

[0017] Figure 3 This is a schematic diagram of the structure in the first state of the present invention.

[0018] Figure 4 This is a schematic diagram of the structure in the second state of the present invention.

[0019] Figure 5 This is a structural schematic diagram of the third state of the present invention.

[0020] Figure 6 This is a side view of the present invention.

[0021] Figure 7 This is a bottom view of the present invention.

[0022] Figure 8 This is a schematic diagram of the usage state of the present invention.

[0023] Figure 9 This is a structural diagram of the connecting box.

[0024] Figure 10 This is a schematic diagram of the internal structure of the connecting box.

[0025] In the diagram: Headset housing-1, First connecting strap-11, Second connecting strap-12, Connecting box-2, Connecting port-21, Drive gear-22, Rack-23, Anti-detachment block-24; Auxiliary fixing strap-3, Telescopic adjustment buckle-4, Connecting wire-5, Electrode plate-51. Detailed Implementation

[0026] The invention will now be further described with reference to the accompanying drawings.

[0027] Please see Figure 1 and Figure 2 As shown, the present invention provides an emotion recognition method based on a multimodal fusion neural network, characterized in that it includes: The VR wearable device body includes a headset shell, a flexible padding that conforms to the face, and a headband. The VR wearable device body can be made of various materials, such as lightweight plastics or composite materials, with the headset shell housing a display screen, optical lenses, and other electronic components. To improve wearing comfort and effectively block external light, a flexible padding that conforms to the face can be attached to the inside of the headset shell. This padding can be made of materials such as sponge, silicone, or fabric and is removable and washable. Furthermore, to securely fix the VR wearable device body to the user's head, a headband can be provided. This headband can use elastic webbing, Velcro, or a simple adjustable buckle structure to accommodate different user head sizes. A multimodal physiological signal acquisition module, integrated into the VR wearable device body, includes: At least four dry electrode EEG acquisition units are deployed at corresponding Fp1 and Fp2 locations in the prefrontal lobe and T3 and T4 locations in the temporal lobe, respectively, to acquire EEG signals. Dry electrode EEG acquisition units are electrodes that can directly contact the skin to acquire EEG signals without the need for conductive gel. They are typically made of Ag / AgCl coating or conductive polymer materials, exhibiting good biocompatibility and conductivity. Deployment at Fp1 and Fp2 locations effectively captures EEG activity related to emotion and cognitive load, while deployment at T3 and T4 locations helps capture signals related to hearing, language, and some emotional processing. The advantage of dry electrodes lies in their ease of wear, reducing the preparation time compared to traditional wet electrodes, making them more suitable for the immediate use scenarios of VR devices.

[0028] The binocular optical eye-tracking unit, composed of an infrared light source and a high-speed camera, is deployed inside the headset below the eye sockets to collect data on pupil diameter changes, blink frequency, and gaze trajectory. This unit tracks eye movements by analyzing the infrared light reflected from the eyeballs, combining the infrared light source and the high-speed camera. The infrared light source typically uses invisible wavelengths to avoid interfering with the user's vision, while the high-speed camera captures eye images at hundreds of frames per second, ensuring tracking accuracy and real-time performance. Its placement inside the headset below the eye sockets ensures effective coverage of both eyes without obstructing the user's field of vision. This eye-tracking data is an important indicator of emotional and cognitive states.

[0029] The photoplethysmography (PPG) sensor consists of two dry electrodes deployed at the temple to acquire photoplethysmography (PPG) pulse wave signals. This sensor obtains heart rate information by measuring changes in blood volume at the skin surface and typically comprises a light-emitting diode (LED) and a photodiode. The LED emits light of a specific wavelength; this light penetrates the skin tissue and is absorbed by the blood, while the unabsorbed light is received by the photodiode. With each heartbeat, the blood volume within the vessels changes periodically, causing a corresponding change in the intensity of the light received by the photodiode, thus forming the pulse wave signal. Deploying it at the temple, an area rich in blood vessels and relatively stable, facilitates signal acquisition. The extracted heart rate variability parameters are important indicators for assessing autonomic nervous system activity.

[0030] The skin conductance sensor consists of two gold-plated dry electrodes deployed on the corresponding cheek area of ​​the flexible pad to collect skin conductivity levels. This sensor measures skin conductivity, reflecting sweat gland activity, which is controlled by the sympathetic nervous system and closely related to emotional arousal. The gold plating provides excellent conductivity and corrosion resistance, ensuring stable signal acquisition. The dry electrodes eliminate the need for applying conductive gel compared to traditional wet electrodes, making them more suitable for quick wear and prolonged use of VR devices. Deploying them on the corresponding cheek area of ​​the flexible pad ensures stable contact and signal acquisition due to the high density of sweat glands and close fit to the VR headset.

[0031] By integrating and optimizing the deployment of specific types of physiological signal acquisition units onto the VR wearable device, the accuracy, comfort, and convenience issues of physiological signal acquisition in a VR environment are effectively resolved. The dry electrode EEG acquisition unit, binocular eye-tracking unit, photoplethysmography sensor, and electrodermal response sensor are each selected with the most suitable sensor type and deployment location based on the characteristics of different physiological modalities. For example, the dry electrode EEG acquisition unit avoids the cumbersome preparation of traditional wet electrodes, improving the user experience; the eye-tracking unit can non-invasively capture detailed ocular physiological data; and the photoplethysmography sensor and electrodermal response sensor, through the use of dry electrodes, ensure comfort and signal stability during prolonged wear. This optimized sensor configuration can synchronously and with high quality acquire multimodal physiological signals, providing rich and reliable input data for subsequent edge computing preprocessing units and multimodal fusion neural network models. This significantly improves the accuracy and real-time performance of emotion recognition, enabling the VR content adaptive adjustment module to respond more accurately to changes in user emotions and provide an immersive and personalized virtual experience.

[0032] An edge computing preprocessing unit, embedded within the VR headset, can be a microcontroller or dedicated chip embedded in the VR wearable device itself. Its functions include digitizing analog signals and performing basic signal processing operations. For example, digital filters can be used to perform low-pass or high-pass filtering to remove power frequency interference or high-frequency noise. Signal slicing can be achieved by setting a fixed time window length and sliding step size, dividing the continuous signal stream into discrete samples. Normalization processing can use simple linear scaling or Z-score normalization methods to adjust signals of different modes or amplitudes to a uniform numerical range, thereby generating multi-channel time-series data with consistent structure; including: Analog-to-digital converters are used to convert analog physiological signals into 24-bit high-precision digital signals; The digital signal processor is equipped with a bandpass filter bank, a motion artifact adaptive filter, and a sliding window slicer to generate time-aligned multi-channel time-series samples. A multimodal fusion neural network model, which can be deployed on the computing chip inside the VR wearable device or transmitted wirelessly to an external computing device for processing; the model consists of the following structure: The spatial feature extraction layer consists of four parallel, structurally independent one-dimensional convolutional neural network branches, corresponding to four modalities: EEG, eye movement, pulse wave, and electrodermal transfer (EDT). Each branch includes at least three convolutional blocks, each consisting of a stack of one-dimensional convolutional layers, batch normalization layers, ReLU activation layers, and max pooling layers. Specifically, the one-dimensional convolutional layers effectively capture local patterns and features from the raw temporal data, such as specific frequency components of EEG or instantaneous changes in eye movement signals. The batch normalization layers normalize the inputs to each layer, accelerating model training and improving stability. The ReLU activation layers introduce nonlinearity, enhancing the model's expressive power. The max pooling layers are used for downsampling, extracting the most important local features and reducing computational cost. This parallel structure ensures that the raw information of each modality can be processed independently and fully in the early stages, avoiding mutual interference between feature extractions of different modalities, thus providing high-quality modality-specific features for subsequent fusion. The temporal feature extraction layer consists of a bidirectional long short-term memory (Bi-LSTM) network layer. Its input is the four modal feature sequences output by the spatial feature extraction module, and its output is a temporal feature vector with forward and backward contextual information. This Bi-LSTM network layer contains 128 hidden units, with an input dimension equal to the sum of the dimensions of the feature vectors output from each modal branch, and an output dimension of 256. Specifically, the Bi-LSTM network is a recurrent neural network particularly adept at processing and predicting time-series data. By combining forward and backward LSTM units, it can simultaneously capture past and future dependencies in the sequence, thereby comprehensively understanding the dynamic patterns and long-term dependencies of physiological signals over time, which is crucial for accurately identifying emotional states. The cross-modal attention fusion layer comprises a multi-head attention mechanism layer and a feature concatenation layer. The multi-head attention mechanism layer uses the temporal feature vector as the query, key, and value input to calculate the attention weight distribution for each modality. The feature concatenation layer concatenates the weighted features from each modality to form a fused feature vector. The multi-head attention mechanism layer employs four attention heads, each with a dimension of 64, and calculates the weight coefficients of each modality feature relative to the emotion recognition task through scaled dot product attention. These weight coefficients are updated in real-time during the model inference phase. Specifically, the multi-head attention mechanism allows the model to focus on different parts of the input sequence in parallel across different "representation subspaces," thereby capturing the complex relationships between modalities more comprehensively and meticulously. By dynamically calculating attention weights, the model can adaptively identify and highlight the modality information that contributes most to the current emotion recognition task; for example, under specific emotions, EEG signals may be more discriminative than dermal conductance signals. This dynamic weighted fusion method effectively solves the problem of uneven importance of information from different physiological modalities and enhances the model's ability to capture complex cross-modal associations. The classification output layer comprises at least two fully connected layers and one Softmax output layer, used to map the fused feature vector to a probability distribution over a preset emotion category. The preset emotion categories include four quadrants: high arousal-high pleasure, high arousal-low pleasure, low arousal-high pleasure, and low arousal-low pleasure, as well as a neutral emotion category, for a total of five output categories. Specifically, the fully connected layers are responsible for converting the fused high-dimensional features into low-dimensional category representations, while the Softmax output layer transforms these representations into a probability distribution for each emotion category, ensuring that the sum of the probabilities of all categories is 1, thereby providing clear and interpretable emotion recognition results. The five emotion output categories cover the arousal-pleasure two-dimensional emotion model, enabling fine-grained emotion recognition and meeting the refined requirements of VR content adaptive adjustment. VR content adaptive adjustment module, including: The emotional state analysis unit is used to receive the probability distribution and output discrete emotional labels and continuous pleasure-arousal coordinate values; and the real-time rendering engine interface is used to dynamically adjust the tone mapping, environmental sound effect parameters, narrative branch weights or interaction difficulty coefficients of the virtual scene according to the emotional labels or coordinate values. The real-time rendering engine interface receives discrete emotion tags or continuous pleasure-arousal coordinates from the emotion state analysis unit and converts them into instructions that the VR rendering engine can recognize and execute. Through this interface, the system can dynamically adjust various presentation parameters of the virtual scene. For example, it can adjust the scene's tone mapping based on the emotional state, such as increasing color saturation and brightness when the user is happy, and decreasing saturation and shifting towards cooler tones when the user is sad; it can adjust environmental sound parameters, such as increasing the rhythm and volume of background music to reflect tension, or playing soothing natural sounds when the user is relaxed; in VR experiences with multi-branch narratives, it can adjust the weight of narrative branches based on the user's emotions, guiding the user into a storyline that better matches their current emotional state; or it can adjust the interaction difficulty coefficient, such as reducing the game difficulty when the user is frustrated, and increasing the challenge when the user is excited. This interface ensures that emotion recognition results can be applied to the presentation of the virtual scene in real time and effectively, achieving a highly personalized VR experience.

[0033] The VR wearable device body refers to the main part of the virtual reality device worn by the user. It typically includes a head-mounted display shell to enclose the internal components, flexible padding that comes into direct contact with the user's face to provide comfort and light protection, and a headband to secure the device to the user's head. This body provides the physical carrier and wearing interface for the entire emotion recognition system.

[0034] A multimodal physiological signal acquisition module refers to a hardware suite used to simultaneously acquire multiple physiological response data from a user. These physiological signals may include electroencephalogram (EEG) signals reflecting brain activity, eye movement data reflecting visual attention and cognitive load, heart rate variability signals reflecting autonomic nervous system activity, and skin conductance signals reflecting emotional arousal levels. This module is typically designed to be lightweight and integrated into the VR wearable device itself to achieve a seamless or minimally invasive wearing experience.

[0035] Edge computing preprocessing units refer to modules deployed on computing devices close to the data source (i.e., the VR wearable device itself) for preliminary processing of raw physiological signals. Their functions include filtering the signal to remove noise, slicing it into analyzable time segments, and normalizing it to eliminate differences between different signal sources or individuals, ultimately generating structured multi-channel time-series data to provide high-quality input for subsequent emotion recognition.

[0036] A multimodal fusion neural network model is a deep learning model specifically designed to process and fuse temporal data from different physiological modalities. This model typically contains multiple layers, such as a spatial feature extraction layer to extract local features from each modality's data, a temporal feature extraction layer to capture dynamic patterns of feature changes over time, a cross-modal attention fusion layer to intelligently integrate information from different modalities and assign weights to each modality, and a classification output layer to map the fused features to specific emotion categories.

[0037] A VR content adaptive adjustment module refers to a software or hardware component that adjusts the presentation of a virtual reality scene in real time based on the identified user's emotional state. This module can dynamically modify parameters such as visual elements (e.g., brightness, color), auditory elements (e.g., background music, sound effects), and interaction logic (e.g., task difficulty, NPC behavior) of the virtual scene according to changes in the user's emotions, in order to enhance the user experience or guide the user's emotions.

[0038] The motion artifact adaptive filter is a signal processing unit that can dynamically adjust its parameters according to the characteristics of the input signal. Its core function is to identify and eliminate noise components caused by user movement in physiological signals. By adjusting the filtering characteristics in real time, it can effectively separate motion artifacts from the target physiological signal, thereby improving the purity and usability of the signal. The real-time motion compensation algorithm based on eye-tracking signals is a method that uses motion information collected by the user's eye-tracking unit to counteract motion artifacts in physiological signals. Its "real-time" characteristic means that the algorithm can process data while it is being acquired, ensuring the immediacy of emotion recognition. The key to this algorithm is to use the acceleration data reflecting head or eye movement in the eye-tracking signal as a reference to help identify and remove interference in the physiological signal. The eye-tracking unit is usually used to monitor the user's gaze direction and pupil changes, but its integrated sensors (such as inertial measurement units) can also synchronously collect acceleration data of the user's head or eyes. Using this acceleration data as a reference signal can provide the adaptive filter with auxiliary information that is highly correlated with motion artifacts, thereby more accurately estimating and eliminating motion interference in the physiological signal. The adaptive least mean square (LMS) algorithm is an iterative optimization algorithm widely used in adaptive filtering. It minimizes the mean square error between the filter's output and the desired signal by continuously adjusting the filter's weight coefficients. In this application, the LMS algorithm utilizes the acceleration reference signal provided by the eye-tracking unit to learn and model the characteristics of motion artifacts in real time, thereby effectively subtracting these artifacts from EEG and pulse wave signals. EEG and pulse wave signals are highly sensitive to motion artifacts. For example, slight head movements can produce significant artifacts in an EEG, while body movement can affect the morphology of the pulse wave. Through the aforementioned adaptive filtering mechanism, signals from these modalities can be specifically optimized to ensure that the extracted physiological features accurately reflect the user's emotional state, rather than noise caused by motion.

[0039] The sliding window slicer is configured with a window length of 3 seconds and a sliding step of 0.5 seconds. Adjacent windows have a 75% overlap. The slicer timestamps each modal signal during sample generation to ensure temporal consistency of multimodal data. Specifically, the sliding window slicer is used to segment a continuous stream of physiological signals into discrete time windows or samples. A window length of 3 seconds means that each independent data sample used for emotion recognition will contain 3 seconds of physiological signal data. Choosing 3 seconds as the window length is based on considerations of the dynamic changes in human emotions. This length captures both instantaneous fluctuations in emotion and contains sufficient information to reflect the persistent characteristics of emotion, avoiding insufficient information due to overly short windows or excessive irrelevant information introduced by overly long windows. The sliding step of 0.5 seconds refers to the time interval between the starting points of two adjacent data windows. A smaller sliding step increases the number of data samples, providing denser training data for the neural network. This helps the model better learn subtle changes in emotion and allows the emotion recognition system to output emotional states at a higher frequency, achieving more real-time emotion monitoring and feedback. The 75% overlap between adjacent windows effectively increases the number of training samples, improving the model's generalization ability. Furthermore, it ensures the continuity of emotional states is preserved in adjacent samples, reducing information loss due to window boundary effects, thus making the model's capture of emotional changes smoother and more accurate. In addition, the slicer timestamps the signals of each modality when generating samples. This ensures that physiological signals from different modalities (such as EEG signals, eye-tracking data, heart rate variability signals, and skin conductance signals) are precisely synchronized on the time axis before slicing them into samples. Since different modal signals may be collected by different sensors at different sampling rates and start times, time calibration must be performed before slicing. This can be achieved by recording the precise timestamp of each data point and interpolating, resampling, or truncating using a unified time reference, ensuring that all modal data within the same time window correspond to physiological activities occurring in the same time period, thereby ensuring the temporal consistency of multimodal data. Temporal consistency refers to the fact that, during multimodal data fusion, all modal data precisely correspond to events or states within the same time period. Through the aforementioned timestamp alignment mechanism, it can be guaranteed that within any 3-second sliding window, EEG signals, eye-tracking data, heart rate variability signals, and skin conductance response signals accurately reflect the user's physiological state within that 3-second period.

[0040] In the four parallel one-dimensional convolutional neural network branches: The convolution kernel size of the EEG signal branch is 3×1, the stride is 1, and the number of output channels is 32, 64, and 128 respectively. The convolution kernel size of the eye-tracking signal branch is 5×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The convolution kernel size of the pulse wave signal branch is 4×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The kernel size of the electrodermal signal branch is 8×1, the stride is 1, and the number of output channels is 8, 16, and 32 respectively. Each branch's pooling layer uses max pooling with a size of 2×1.

[0041] In this study, the EEG signal branch used a 3×1 convolutional kernel size, a stride of 1, and output channels of 32, 64, and 128 respectively. EEG signals typically contain rich time-frequency information, and their characteristics often manifest as localized, subtle fluctuations. A smaller convolutional kernel size (3×1) effectively captures these high-frequency, short-term local features, avoiding information loss due to excessive smoothing. A stride of 1 ensures a comprehensive scan of the signal. The progressively increasing number of output channels (32, 64, 128) allows the network to gradually extract more abstract and discriminative feature representations from the raw EEG signal, such as rhythmic activity or event-related potentials in different frequency bands.

[0042] The eye-tracking signal branch uses a 5×1 convolutional kernel with a stride of 1, and output channels of 16, 32, and 64 respectively. Eye-tracking data, such as pupil diameter changes, blink frequency, and gaze trajectory, typically fall between EEG and TENS in terms of time scale, reflecting physiological responses at a moderate time scale. Therefore, using a moderately sized convolutional kernel (5×1) effectively captures broader local patterns such as blink duration and gaze drift. A stride of 1 ensures the continuity of feature extraction. Increasing the number of output channels (16, 32, 64) helps the network learn eye-tracking behavioral patterns related to emotional states.

[0043] The pulse wave signal branch uses a 4×1 convolution kernel with a stride of 1, and output channels of 16, 32, and 64 respectively. The photoplethysmography (PPG) pulse wave signal exhibits significant periodicity, and its waveform morphology, amplitude, and frequency variations are closely related to heart rate variability. A 4×1 kernel size effectively covers the typical morphology of a single pulse wave, thus efficiently extracting features related to heart rate and heart rate variability. A stride of 1 ensures the capture of pulse wave details. Increasing the number of output channels (16, 32, 64) helps the network learn deeper levels of cardiovascular activity characteristics.

[0044] The EDS signal branch uses an 8×1 kernel size, a stride of 1, and output channels of 8, 16, and 32 respectively. EDS signals typically exhibit slowly changing baseline conductance and occasional, prolonged skin response (SCR). A larger kernel size (8×1) is better suited for capturing these prolonged, relatively slow-changing signal features, such as trends in skin conductance or the rising and falling edges of the SCR. A stride of 1 ensures complete extraction of these slowly changing features. Increasing the number of output channels (8, 16, 32) enables the network to extract features related to autonomic nervous system activity from EDS signals.

[0045] Furthermore, all pooling layers in each branch employ 2×1 max pooling. Max pooling downsamples the feature map, effectively reducing feature dimensionality while preserving the most salient features, thus reducing computational cost and enhancing the model's robustness to small shifts in the input signal. The 2×1 pooling operation achieves appropriate feature compression while maintaining key information.

[0046] The bidirectional long short-term memory network layer contains 128 hidden units. Its input dimension is the sum of the feature vector dimensions output by each modality branch, and its output dimension is 256.

[0047] The multi-head attention mechanism layer uses four attention heads, each with a dimension of 64. It calculates the weight coefficients of each modality feature relative to the emotion recognition task by scaling dot product attention, and the weight coefficients are updated in real time during the model inference stage.

[0048] Please see Figures 3 to 10 As shown, this application further proposes an optimized head-mounted support structure. Specifically, a first connecting strap 11 and a second connecting strap 12 are respectively connected to the left and right ends of the head-mounted display shell 1. The head-mounted display shell 1 is the main structure of the VR device, housing core components such as the display screen and optical components. The first connecting strap 11 and the second connecting strap 12 are flexible or semi-flexible structures used to fix the head-mounted display shell 1 to the user's head, typically made of fabric, plastic, or composite materials, designed to provide basic wearing support. The ends of the first connecting strap 11 and the second connecting strap 12 are connected via a connecting housing 2. The connecting box 2 has a connecting port 21 at both the front and rear ends. A drive motor (not shown) is embedded in the left and right ends of the connecting port 21. The output end of the drive motor is connected to a drive gear 22. The outer ends of the first connecting belt 11 and the second connecting belt 12 are provided with racks 23 that mesh with the drive gear 22. The ends of the first connecting belt 11 and the second connecting belt 12 are provided with anti-detachment blocks 24. An auxiliary fixing belt 3 is provided between the middle of the head-mounted display shell 1 and the connecting box 2. The auxiliary fixing belt 3 is provided with a telescopic adjustment buckle 4.

[0049] The connecting box is a structural component used to gather and secure the ends of the first and second connecting straps. It is typically located behind the user's head, serving as a connection and support, and may integrate an adjustment mechanism. Both the front and rear ends of the connecting box have connecting openings that allow the connecting straps or their racks to pass through or enter for length adjustment or engagement with internal mechanisms. Drive motors are embedded at both ends of each connecting opening. These drive motors are electromechanical devices that provide the power source, typically small DC motors or stepper motors. They are embedded near the connecting openings inside the connecting box to drive the movement of the connecting straps, enabling automatic or semi-automatic tension adjustment. The output end of the drive motor is connected to a drive gear, a mechanical transmission component connected to the drive motor's output shaft. This drive gear, typically a cylindrical or bevel gear, converts the motor's rotational motion into linear motion or further transmits it to other gears to drive the rack on the connecting strap. Both the first and second connecting straps have racks at their outer ends that mesh with the drive gear. The racks are straight rods with toothed structures, typically made of plastic or metal. They form a gear-rack transmission pair with the drive gear. When the drive gear rotates, the racks move along their length, thereby changing the effective length of the connecting straps and adjusting the tightness of the headset. The ends of both the first and second connecting straps have anti-slip blocks, which are protrusions or thickened structures at the ends of the connecting straps. These blocks prevent the connecting straps from completely detaching from the connecting housing or drive mechanism during adjustment, ensuring wearing safety. Furthermore, an auxiliary fixing strap is provided between the middle of the headset shell and the connecting housing. This auxiliary fixing strap is an additional fixing structure besides the first and second connecting straps. It typically extends from the top of the headset shell to the connecting housing, crossing over the user's head. It provides additional vertical support and stability, preventing the headset from sagging or wobbling. The auxiliary fixing strap is equipped with a telescopic adjustment buckle, which is an adjustable component on the auxiliary fixing strap. It is usually composed of a buckle, a sliding buckle, or Velcro, and allows users to manually or automatically adjust the length of the auxiliary fixing strap to adapt to different head curvatures, further optimizing the comfort and stability of wearing it.

[0050] Through the above technical solutions, the head-mounted fixing structure of the VR wearable device has been significantly optimized. The cooperation between the drive motor and the drive gears and racks allows for precise and convenient adjustment of the tightness of the first and second connecting straps, even enabling automated adjustment, thus ensuring that the headset shell can stably fit the heads of different users. The anti-slip block design further enhances the safety of wearing the device. In addition, the auxiliary fixing strap and its telescopic adjustment buckle provide additional vertical support, effectively preventing the headset from falling or wobbling, further enhancing the stability and comfort of wearing the device. This stable and adjustable wearing method not only improves the user experience, but more importantly, it ensures continuous and effective contact between the multimodal physiological signal acquisition module and the user's skin, reducing signal artifacts caused by improper wearing. This provides high-quality raw data for the subsequent edge computing preprocessing unit and multimodal fusion neural network model, ultimately improving the accuracy and reliability of emotion recognition.

[0051] In some embodiments described above, the VR wearable device achieves stable wearing of the user's head through structures such as a headset shell, flexible padding, and headband, and integrates a multimodal physiological signal acquisition module to acquire the user's physiological data. However, in practical applications, ensuring that the weak electrical signals acquired by these physiological signal acquisition units distributed throughout the VR wearable device can be stably and reliably transmitted to the edge computing preprocessing unit, while avoiding external interference and connection loosening or damage caused by user activity, is a key challenge in achieving high-precision emotion recognition.

[0052] Please see Figures 3 to 10 As shown, this application further proposes that the head-mounted display housing 1 is connected to a plurality of connecting wires 5, and the ends of the connecting wires 5 are connected to electrode plates 51.

[0053] Specifically, the headset housing is connected to multiple connecting wires, which are used to transmit physiological signals acquired from the multimodal physiological signal acquisition module integrated into the VR wearable device. These wires are designed to be tightly integrated with the headset housing and can be connected in various ways, such as through pre-drilled wiring channels inside the headset housing or by integrating them into the housing structure using a flexible printed circuit board (FPC), thus achieving concealment and interference resistance of the signal transmission path. This integration method helps protect the wires from external physical damage and reduces pulling and wear caused by user wear or movement, thereby ensuring the stability of signal transmission. Simultaneously, electrode pads are connected to the ends of the connecting wires. These electrode pads serve as the physical and electrical interface between the connecting wires and the actual physiological signal sensors (e.g., dry electrodes of the EEG acquisition unit, gold-plated dry electrodes of the skin conductance sensor, etc.). These electrode pads can be reliably connected to the ends of the connecting wires using methods such as soldering, crimping, conductive adhesive bonding, or spring contacts, ensuring a low-impedance signal path. The electrode pads should be designed for compatibility with physiological sensors. For example, they can be designed as conductive pads that directly contact dry electrodes, or as part of a flexible circuit board for easy integration with the sensor module. Through this connection, the weak electrical signals acquired by the physiological sensor can be efficiently and losslessly transmitted via connecting wires to subsequent processing units.

[0054] By employing the aforementioned technical solution, multiple connecting wires are attached to the outer shell of the headset, with electrode pads connected to the ends of these wires. This application effectively solves the reliability problem of physiological signal transmission within the VR wearable device. This design embeds or tightly integrates the signal transmission path into the headset shell structure, significantly reducing the clutter and potential physical damage risks associated with external wiring. The robust connection between the connecting wires and electrode pads ensures that raw signals collected from various physiological sensors, such as EEG signals, eye-tracking data, heart rate variability signals, and skin conductance signals, can be transmitted with high fidelity and low noise. This not only improves the signal acquisition quality and stability of the entire emotion recognition system, avoiding signal loss or distortion due to poor connections, but also enhances the overall durability of the VR device and user comfort. It provides high-quality input data for subsequent edge computing preprocessing units and multimodal fusion neural network models, thereby ensuring the accuracy and real-time performance of emotion recognition.

[0055] The preset emotion categories include four quadrants: high arousal-high pleasure, high arousal-low pleasure, low arousal-high pleasure, and low arousal-low pleasure, as well as a neutral emotion category, for a total of five output categories.

[0056] Step A: Collect multimodal physiological signals of the user in the VR environment, including electroencephalogram (EEG) signals, eye movement signals, photoplethysmography (PPG) signals, and skin conductance signals; Step B: Preprocess the acquired signals, including: B1: A 0.5-50Hz bandpass filter was used to filter the EEG signal, a 0.5-5Hz bandpass filter was used to filter the pulse wave signal, and a 0.05-10Hz bandpass filter was used to filter the skin conductance signal. B2: An adaptive filtering algorithm based on eye-tracking data is used to remove motion artifacts; B3: Perform sliding window slicing on the filtered signal to generate time-aligned multi-channel time series samples; B4: Perform Z-score normalization on each sample; Step C: Input the preprocessed multimodal time series samples into the pre-trained multimodal fusion neural network model; Step D: In the spatial feature extraction module, each modal signal is processed through an independent one-dimensional convolutional neural network branch to extract local spatial features and generate intramodal feature maps; Step E: In the temporal feature extraction module, the intramodal feature maps of each modality are input into the bidirectional long short-term memory network layer according to the time step, the forward and backward temporal dependencies are extracted, and the temporal feature vector is output. Step F: In the cross-modal attention fusion module, the attention weights of the temporal feature vectors of each modality are calculated through a multi-head attention mechanism, and the weighted feature vectors are concatenated to generate a fused feature vector; Step G: In the classification output module, the fused feature vector is mapped to the emotion category space through a fully connected layer, and the probability of each category is output through the Softmax function; Step H: Based on the output emotion category probability, drive the VR content adaptive adjustment module to adjust the presentation parameters of the virtual scene in real time.

[0057] In step B1, a fifth-order Butterworth bandpass filter is used to filter the EEG signal, and an additional 50Hz notch filter is added to eliminate power frequency interference.

[0058] Multimodal physiological signals were collected from at least 100 subjects while they watched standardized emotional stimuli, and a labeled dataset was constructed by combining subject self-assessment with expert annotation. End-to-end supervised training is performed using a joint loss function, which is expressed as follows:

[0059] Among them, L ce For cross-entropy classification loss, L cl To compare the learning loss, 𝜆1 and 𝜆2 are balancing coefficients; The contrastive learning loss L clIt is used to bring different modal features within the same emotion category closer together in the feature space, and to widen the distance between features within different emotion categories; The Adam optimizer was used, with an initial learning rate of 0.001, a batch size of 64, and 200 training epochs. Early stopping was employed to prevent overfitting.

[0060] When a user first uses the device, baseline physiological signals of the user at rest are collected for 1 minute. Ten images from a standardized emotional image library were presented, and the user's physiological signal response to the above stimuli was collected. Using the data collected above, the parameters of the fully connected layer of the classification output module of the multimodal fusion neural network model are fine-tuned through transfer learning, and the parameters of the spatial feature extraction module and the temporal feature extraction module are frozen.

[0061] The following provides a detailed technical description of the neural network model, signal processing method, and training strategy in the core technical solution to ensure that those skilled in the art can implement it accordingly.

[0062] The detailed structural design of each branch is shown in the table below: Branches Layer type convolution kernel Step length Input Channel Output Channel Output size EEG branches (8 channels) Conv1D + BN + ReLU 3×1 1 8 32 T×32 EEG branches (8 channels) MaxPool1D 2×1 2 32 32 T / 2×32 EEG branches (8 channels) Conv1D + BN + ReLU 3×1 1 32 64 T / 2×64 EEG branches (8 channels) MaxPool1D 2×1 2 64 64 T / 4×64 EEG branches (8 channels) Conv1D + BN + ReLU 3×1 1 64 128 T / 4×128 EEG branches (8 channels) MaxPool1D 2×1 2 128 128 T / 8×128 Eye movement branch (4 channels) Conv1D + BN + ReLU 5×1 1 4 16 T×16 Eye movement branch (4 channels) MaxPool1D 2×1 2 16 16 T / 2×16 Eye movement branch (4 channels) Conv1D + BN + ReLU 5×1 1 16 32 T / 2×32 Eye movement branch (4 channels) MaxPool1D 2×1 2 32 32 T / 4×32 Eye movement branch (4 channels) Conv1D + BN + ReLU 5×1 1 32 64 T / 4×64 Eye movement branch (4 channels) MaxPool1D 2×1 2 64 64 T / 8×64 Pulse wave branch (2 channels) Conv1D + BN + ReLU 4×1 1 2 16 T×16 Pulse wave branch (2 channels) MaxPool1D 2×1 2 16 16 T / 2×16 Pulse wave branch (2 channels) Conv1D + BN + ReLU 4×1 1 16 32 T / 2×32 Pulse wave branch (2 channels) MaxPool1D 2×1 2 32 32 T / 4×32 Pulse wave branch (2 channels) Conv1D + BN + ReLU 4×1 1 32 64 T / 4×64 Pulse wave branch (2 channels) MaxPool1D 2×1 2 64 64 T / 8×64 Electrodermal branch (1 channel) Conv1D + BN + ReLU 8×1 1 1 8 T×8 Electrodermal branch (1 channel) MaxPool1D 2×1 2 8 8 T / 2×8 Electrodermal branch (1 channel) Conv1D + BN + ReLU 8×1 1 8 16 T / 2×16 Electrodermal branch (1 channel) MaxPool1D 2×1 2 16 16 T / 4×16 Electrodermal branch (1 channel) Conv1D + BN + ReLU 8×1 1 16 32 T / 4×32 As shown in the table above, after each branch outputs, global average pooling is performed to compress the time dimension of each modality to 1, resulting in a fixed-length feature vector: 128-dimensional EEG branch, 64-dimensional eye-tracking branch, 64-dimensional pulse wave branch, and 32-dimensional skin conductance branch. Thus, the spatial feature extraction module outputs feature vectors for four modalities.

[0063] The feature vectors of the four modalities are concatenated according to time steps. (Note: The spatial feature extraction described above has already compressed the temporal dimension, so the temporal modeling method needs to be reconsidered here. A more accurate design is to retain the time steps after the spatial convolutional layer and before the pooling layer completely compresses the temporal dimension, and then input them into the Bi-LSTM.) The revised design: After the last convolutional layer, each branch does not perform global average pooling, but instead retains the temporal dimension, outputting a shape of [T / 8 × feature dimension]. Then, the four modalities are concatenated along the feature dimension to form a temporal feature sequence of shape [T / 8 × (128+64+64+32)] = [47 × 288] (because T=375, after three pooling operations with a stride of 2, the temporal dimension becomes 375 / 8≈47). This sequence is then fed into a Bi-LSTM layer.

[0064] Bi-LSTM layer parameters: Number of hidden units: 128 Number of layers: 2 (stacked) Direction: Two-way Output: The outputs of each time step are concatenated into a 256-dimensional vector (128×2). Finally, the output of the last time step is taken as the temporal feature vector with a dimension of 256.

[0065] Input: Temporal feature vectors of four modalities (extracted separately before input to the Bi-LSTM, or fused after independent processing by each branch of the Bi-LSTM). This design employs feature-level fusion followed by attention or modality-level attention.

[0066] This method employs a multi-head attention mechanism for modality-level dynamic weighting: Let the feature vectors of the four modalities be heeg, heye, hppg, and heda, with dimensions of 128, 64, 64, and 32, respectively.

[0067] First, a linear transformation is used to map each mode to the same dimension d. k =64:

[0068]

[0069] Multi-head attention calculation:

[0070]

[0071]

[0072] Self-attention is used here, i.e., Q=K=V=H, with each head having a dimension of 16, for a total of 4 heads.

[0073] The output is Then feature splicing is performed; This yields a 256-dimensional fused feature vector.

[0074] 2.4 Classification Output Module Fully connected layer 1: 256 → 128, Dropout rate 0.5; Fully connected layer 2: 128 → 64, Dropout rate 0.5; Output layer: 64 → 5 (emotion category), Softmax activation.

[0075] At least 100 participants (aged 18-60, gender balanced) were recruited.

[0076] Stimulating materials: International Emotional Image System (IAPS) and self-built VR emotional scenes (such as tranquil forests, roller coasters, haunted castles, cozy homes, etc.).

[0077] Labeling: After each stimulus segment, subjects used the Self-Assessment Manikin scale to rate pleasure (1-9) and arousal (1-9), and mapped the scores to four quadrants (high pleasure and high arousal, high pleasure and low arousal, low pleasure and high arousal, low pleasure and low arousal), and added a neutral category (pleasure 4.5-5.5 and arousal 4.5-5.5).

[0078] The loss function is designed as follows:

[0079] Where 𝜆1=1, 𝜆2=0.5.

[0080] Cross-entropy loss:

[0081] Where N is the batch size, C=5 is the number of categories, and y is the one-hot encoding of the actual label. Predict probabilities for the model.

[0082] Contrastive learning loss (used to enhance intermodal consistency): Within a batch, for each sample, we treat its four modal feature vectors (i.e., the modal features heeg, heye, hppg, and heda before the Bi-LSTM input) as different views of the same instance. Definition: Where z represents the features mapped by the projection head (a two-layer MLP), sim is the cosine similarity, and τ=0.07 is the temperature coefficient. This loss forces features from different modalities within the same emotion category to move closer together, while features from different samples move further apart.

[0083] Optimizer: Adam, β2=0.999; β2=0.999 Initial learning rate: 0.001, decayed to 1e-5 using cosine annealing. Batch size: 64 Maximum number of rounds: 200, early stop method (stop if losses do not decrease for 10 consecutive rounds). Validation set partitioning: 80% training set, 10% validation set, and 10% test set. Model saving: Select the model with the highest accuracy on the validation set as the final model.

[0084] Freeze all parameters of the spatial feature extraction module and the temporal feature extraction module.

[0085] Only the classification output module (two fully connected layers) was fine-tuned.

[0086] Use 20-30 calibration samples collected by the user, set the learning rate to 0.0001, and train for 10-20 rounds.

[0087] This embodiment provides an emotion recognition system based on a multimodal fusion neural network and a VR wearable device. The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings.

[0088] I. System Hardware Configuration and Signal Acquisition 1.1 VR Wearable Device Main Body The VR wearable device used in this embodiment is a customized head-mounted display device, including: Headset housing: Made of lightweight ABS material, weighing approximately 380g; Flexible padding: conforms to facial contours, made of medical-grade silicone, 3mm thick, with embedded sensor electrodes; Headband: Adjustable tightness to ensure stable wear.

[0089] 1.2 Integrated Layout of Multimodal Physiological Signal Acquisition Module Reference Figure 8 (Layout diagram of the multimodal physiological signal acquisition module on the VR headset), the specific deployment locations of each sensor are as follows: EEG acquisition unit: 4 dry electrodes, located at: Fp1 location: Left forehead, 2cm to the left of the center of the eyebrows; Fp2 location: Right forehead, 2cm to the right of the center of the eyebrows; T3 location: Left temporal lobe, 3cm above the left ear; T4 location: Right temporal lobe, 3cm above the right ear.

[0090] The reference electrode is placed behind the ear at the mastoid process, with a sampling rate of 250Hz and a 16-bit ADC resolution.

[0091] Eye-tracking unit: binocular optical tracking, infrared light source wavelength 850nm, high-speed camera sampling rate 120Hz, output pupil diameter (accuracy 0.1mm), blink events (0 / 1 markers), and gaze coordinates (x,y).

[0092] Photoplethysmography (PPG) sensor: Dual-wavelength (660nm / 940nm) reflective PPG sensor, deployed in the left temple pad, with a sampling rate of 125Hz, used to extract heart rate and heart rate variability features.

[0093] Skin conductance sensor: Two gold-plated dry electrodes, 8 mm in diameter and 2 cm apart, deployed on the right cheek pad, sampling rate 50 Hz, DC coupling mode, measurement range 0-20 μS.

[0094] Accelerometer: Integrated into the eye-tracking unit, a three-axis MEMS accelerometer with a range of ±8g and a sampling rate of 200Hz, used as a reference signal for adaptive filtering of motion artifacts.

[0095] 1.3 Edge Computing Preprocessing Unit The STM32H747 dual-core processor (Cortex-M7@480MHz + Cortex-M4@240MHz) embedded inside the VR headset has the following configuration: Analog-to-digital converter: 24-bit Σ-Δ ADC, synchronous sampling; Digital signal processor: Hardware supports FIR / IIR filtering and FFT operations; Memory: 2MB SRAM for data buffering.

[0096] II. Signal Preprocessing and Motion Artifact Removal 2.1 Filtering and Resampling Taking a 10-second data segment as an example (the user's head sways from side to side at a frequency of approximately 1Hz after wearing the device), the raw signal undergoes the following preprocessing steps: EEG signal: A fifth-order Butterworth bandpass filter (0.5-50Hz) is used to eliminate baseline drift and high-frequency noise, and an additional 50Hz notch filter is used to eliminate power frequency interference.

[0097] Pulse wave signal: 0.5-5Hz bandpass filter, retaining heart rate-related frequency bands.

[0098] Electrodermal signal: 0.05-10Hz bandpass filter, preserving tetanic and phase components.

[0099] Time alignment and resampling: Cubic spline interpolation is used to resample all signals to 125Hz to ensure the timing consistency of multimodal data.

[0100] 2.2 Adaptive Filtering of Motion Artifacts Based on Eye-Tracking Signals Taking the Fp1 channel of EEG signals as an example, the specific implementation process is as follows: Step 1: Construct the reference signal vector The synchronously acquired triaxial acceleration data are: a_x(n) = 0.12g, a_y(n) = 0.08g, a_z(n) = 0.05g (with the head slightly turned to the right). A 7-dimensional reference vector is constructed: Acceleration amplitude: a_mag = sqrt(0.12² + 0.08² + 0.05²) = 0.15g Differences along each axis: a_dot_x = a_x(n) - a_x(n-1) = 0.12 - 0.11 = 0.01g Similarly, calculate a_dot_y and a_dot_z. Finally, x(n) = [0.12, 0.08, 0.05, 0.15, 0.01, 0.00, -0.01]ᵀ Step 2: Filter Configuration An NLMS adaptive filter with order M=48, step size μ=0.03, and regularization parameter δ=1e-5 is used.

[0101] Step 3: Filtering and Updating The input vector X(n) is composed of the current and the previous 47 time steps x(n) (48×7=336 dimensions).

[0102] The filter output y(n) = Wᵀ(n)X(n) is used to estimate the motion artifact components.

[0103] The error signal e(n) = d(n) - y(n), where d(n) is the original EEG signal containing artifacts.

[0104] Normalized step size: μ_eff = 0.03 / (||X(n)||² + 1e-5) Weight update: W(n+1) = W(n) + μ_eff·e(n)·X(n) Step 4: Output and Effects After adaptive filtering, the output e(n) is the clean EEG signal after removing motion artifacts. Motion artifacts are effectively suppressed, and the alpha wave (8-12Hz) is clearly visible. The signal-to-noise ratio improvement is calculated as follows: The original signal SNR is 5.2 dB. Filtered SNR = 19.4 dB Improved by 14.2 dB 2.3 Sliding Window Slicing and Normalization Slice the filtered multimodal signal into a sliding window: Window length: 3 seconds (375 sampling points, 125Hz) Sliding step size: 0.5 seconds (62.5 sampling points) Overlap rate: Approximately 83% For each sample generated, Z-score normalization is performed on each channel of each modality: xnorm=x−μσxnorm=σx−μ Where μ and σ are the mean and standard deviation of the channel within the window.

[0105] After the above processing, approximately 2 samples are generated per second. Each sample is a multi-channel time series matrix with dimensions of [375 × (8+4+2+1+3)] = [375 × 18] (including 8 channels of EEG, 4 channels of eye tracking, 2 channels of PPG, 1 channel of EDA, and 3 channels of acceleration).

[0106] III. Multimodal Fusion Neural Network Model 3.1 Model Structure The specific network structure used in this embodiment is as follows: 3.1.1 Spatial Feature Extraction Module (One-Dimensional CNN Branch) Taking EEG branches as an example: Input: [375×8] Conv1: 3×1 kernel, stride 1, 32 output channels → [375×32] MaxPool: 2×1 pooling kernels, step size 2 → [187×32] Conv2: 3×1 kernel, stride 1, 64 output channels → [187×64] MaxPool: 2×1 pooling kernels, step size 2 → [93×64] Conv3: 3×1 kernel, stride 1, 128 output channels → [93×128] MaxPool: 2×1 pooling kernels, step size 2 → [46×128] Output shape: [46×128] (Time dimension preserved for use by Bi-LSTM) Eye movement branch output [46×64], pulse wave branch output [46×64], skin conductance branch output [46×32].

[0107] 3.1.2 Temporal Feature Extraction Module (Bi-LSTM) The outputs of the four branches are concatenated along the channel dimension: 46×(128+64+64+32)=46×288.

[0108] Input Bi-LSTM layer: Number of hidden units: 128 Number of layers: 2 (stacked) Direction: Two-way Output: Each time step outputs a 256-dimensional (128×2) vector. The output of the last time step is taken as the temporal feature vector with a dimension of 256.

[0109] 3.1.3 Cross-modal attention fusion module (multi-head attention) The specific implementation is as follows: Input: Temporal feature vectors of four modalities (actually fused using Bi-LSTM, but here the design is to process the four modal features independently before fusion; for clarity, this embodiment uses Bi-LSTM to output a single feature vector and then does not perform cross-modal attention. If modal-level fusion is used, the steps are as follows:) Another implementation: Input the spatial feature maps (preserving the time dimension) output by each modality's CNN branch into an independent Bi-LSTM to obtain the temporal feature vectors h_eeg (128-dimensional), h_eye (64-dimensional), h_ppg (64-dimensional), and h_eda (32-dimensional) of the four modalities, and then input them into multi-head attention.

[0110] Linear mapping: The features of each modality are mapped to 64 dimensions through a fully connected layer to form a matrix. .

[0111] Multi-head attention: Employs 4 attention heads, each with 16 dimensions.

[0112] Head 1: Calculate Q1=H W1^Q, K1=H W1^K, V1=H W1^V, and output head1=softmax(Q1K1ᵀ / √16)V1 Similar to calculating head2, head3, and head4 Concatenation and Projection: Concat(head1,...,head4)∈ℝ^{4×64}, through projection matrix W^O∈ℝ^{64×64}, output, Feature concatenation: Flatten(H_att) yields a 256-dimensional fused feature vector.

[0113] 3.1.4 Classification Output Module Fully connected layer 1: 256 → 128, ReLU activation, Dropout = 0.5 Fully connected layer 2: 128→64, ReLU activation, Dropout=0.5 Output layer: 64→5, Softmax activation The output categories are: High Arousal - High Pleasure, High Arousal - Low Pleasure, Low Arousal - High Pleasure, Low Arousal - Low Pleasure, and Neutral.

[0114] 3.2 Model Training 3.2.1 Training Dataset Multimodal physiological signals were collected from 120 subjects (aged 20-55 years, half male and half female) in a VR environment: Stimulating materials: Includes 40 VR scene clips, each lasting 15-30 seconds, covering types such as tranquil forests, roller coasters, haunted castles, and cozy homes.

[0115] Labeling: After each segment, participants used the Self-Assessment Manikin scale to rate pleasure (1-9) and arousal (1-9), and the ratings were mapped to five emotion labels.

[0116] Data scale: A total of 48,000 valid samples were collected (approximately 400 samples per subject).

[0117] 3.2.2 Training Configuration Optimizer: Adam, β1=0.9, β2=0.999 Initial learning rate: 0.001, decayed to 1e-5 using cosine annealing. Batch size: 64 Loss function: Joint loss L = L_ce + 0.5·L_cl Cross-entropy loss L_ce is used for classification. Contrastive learning loss L_cl is used to enhance intermodal consistency. Training rounds: 200 rounds, early stop method (stop if the loss does not decrease for 10 consecutive rounds). Hardware platform: NVIDIA RTX 4090 GPU, training time approximately 12 hours 3.2.3 Training Results After training, the results were evaluated on a test set (20 subjects, approximately 8,000 samples): Overall accuracy: 92.7% (±1.5%) Accuracy rates for each category: high-pleasure high-arousal 91.2%, high-pleasure low-arousal 93.5%, low-pleasure high-arousal 90.8%, low-pleasure low-arousal 94.1%, and neutral 92.3%; this verifies the effectiveness of multi-head attention and multimodal fusion.

[0118] IV. User Adaptive Calibration 4.1 Calibration Procedure New users should follow these steps when using the service for the first time: Step 1: Resting state data acquisition (1 minute) The user sat with eyes closed, and baseline EEG alpha wave power, heart rate variability, and skin conductance levels were collected.

[0119] Individual baseline parameters were calculated as follows: mean alpha wave power 12.5 μV², mean heart rate 72 bpm, and skin conductance 2.3 μS.

[0120] Step 2: Standardized emotional stimulation (approximately 5 minutes) Play 30 images from the IAPS image library in sequence (10 positive, 10 neutral, and 10 negative).

[0121] Each image is displayed for 6 seconds, followed by a 3-second blackout break.

[0122] Simultaneous acquisition of multimodal physiological signals.

[0123] Step 3: User self-assessment and labeling After each image is displayed, users can rate the level of enjoyment and arousal on a scale of 1 to 9 using the controller buttons.

[0124] The ratings were converted into five emotion labels.

[0125] Step 4: Fine-tuning of transfer learning Load the pre-trained model (trained on a dataset of 120 subjects).

[0126] Freeze all parameters of the spatial feature extraction module and the temporal feature extraction module.

[0127] Only the classification output module (two fully connected layers) was fine-tuned.

[0128] Use 30 calibration samples (10 positive, 10 neutral, and 10 negative) with a learning rate of 0.0001 for 10 training rounds.

[0129] Step 5: Calibration Verification The calibration effect was tested using the 5 reserved verification images.

[0130] Accuracy before calibration: 84.2%; accuracy after calibration: 91.5%, an improvement of 7.3%.

[0131] 4.2 Calibration Results Tests with 10 new users showed that: Average accuracy before calibration: 83.6% Average accuracy after calibration: 91.2% Average improvement: 7.6% V. VR Content Adaptive Adjustment 5.1 Analysis of Emotional State Take a user's actual experience as an example: Model output probability distribution: [0.02, 0.88, 0.05, 0.03, 0.02] Identified as belonging to the "high arousal-high pleasure" category, with a confidence level of 88%. Mapped to a continuous space: Pleasure = 7.8 / 9, Arousal = 8.2 / 9 5.2 Content Adjustment Strategy Based on the recognition results, the VR content adaptive adjustment module performs the following operations: Tone mapping: The original scene is grayish-blue (lower happiness level), adjusted to orange-red (increased happiness level), tone offset ΔH = +30°.

[0132] Ambient sound effects: The original background music was a soothing piano piece, which was switched to more rhythmic electronic music based on the high arousal state, and the BPM was increased from 60 to 110.

[0133] Narrative Branch: In story-driven games, the current scene is the "castle entrance". In the state of high excitement and high call, the "brave confrontation" branch is automatically selected instead of "cautiously detour".

[0134] Difficulty level: Increase the game difficulty from "Normal" to "Hard" to match the user's high arousal state.

[0135] 5.3 Real-time verification The end-to-end latency test results of this embodiment are as follows: Signal acquisition to preprocessing: 5.2ms Model inference (edge ​​computing): 28.6ms Content rendering adjustment: 4.4ms Total latency: 38.2ms (average), far below the 50ms real-time interaction requirement.

[0136] VI. Performance Verification and Comparison 6.1 Recognition accuracy in different VR environments Tested in four typical VR scenarios: Tranquil Forest: 91.8% Roller coaster: 88.5% (High accuracy rate despite strong motion interference) Horror Castle: 89.2% Cozy Home: 93.1% VII. Summary of the Effects of the Examples This embodiment provides an emotion recognition system based on a multimodal fusion neural network and a VR wearable device, which achieves high-precision, low-latency, and highly robust emotion recognition through the following innovations: Multimodal signal fusion: EEG, eye movement, heart rate, and skin conductance are complementary, overcoming the limitations of single-modality in VR environment; Motion artifact adaptive filtering: Using eye-tracking unit acceleration data, motion interference is effectively filtered out through the NLMS algorithm, improving the signal-to-noise ratio by 14.2dB; Hybrid neural network architecture: CNN extracts spatial features, Bi-LSTM models temporal dependencies, and multi-head attention dynamically fuses cross-modal information, achieving a recognition accuracy of 92.7%; Edge computing and low latency: End-to-end latency of 38.2ms meets the requirements of real-time VR interaction; User-adaptive calibration: Transfer learning fine-tuning eliminates individual differences, improving accuracy for new users by 7.6%.

[0137] This embodiment can be widely applied to VR games, psychotherapy, education and training, immersive social interaction and other fields, significantly improving the level of emotional intelligence in human-computer interaction.

[0138] The following example will provide a more detailed explanation of the above technical solution: Suppose user A is experiencing a virtual reality scenario designed to induce and assess the user's emotional state through immersive interaction.

[0139] User A wears the VR wearable device. This device includes a headset shell, a flexible pad that conforms to the face, and a headband. A first connecting strap and a second connecting strap are connected to the left and right ends of the headset shell, respectively. The ends of the first and second connecting straps are connected via a connecting box. The connecting box has openings at both the front and rear ends, and drive motors are embedded in the left and right ends of these openings. The output ends of the drive motors are connected to drive gears. The outer ends of both the first and second connecting straps have racks that mesh with the drive gears, and anti-detachment blocks are located at the ends of both straps. An auxiliary fixing strap is located between the middle of the headset shell and the connecting box, and this strap has a telescopic adjustment buckle. This structure allows the VR wearable device to be securely and comfortably fixed to the user's head, ensuring the stability of physiological signal acquisition. Multiple connecting wires are connected to the top of the headset, and electrode pads are connected to the ends of these wires. These electrode pads are part of the multimodal physiological signal acquisition module.

[0140] A multimodal physiological signal acquisition module is integrated into the VR wearable device, simultaneously acquiring user A's physiological signals. Specifically, at least four dry electrode EEG acquisition units are deployed at user A's frontal lobe (Fp1, Fp2) and temporal lobe (T3, T4) locations to acquire EEG signals. A binocular eye-tracking unit, consisting of an infrared light source and a high-speed camera, is deployed inside the headset below the eye sockets to acquire pupil diameter changes, blink frequency, and gaze trajectory. A photoplethysmography sensor, consisting of two dry electrodes, is deployed at user A's temples to acquire photoplethysmography (PPG) signals. A skin conductance sensor, consisting of two gold-plated dry electrodes, is deployed on a flexible pad corresponding to one side of user A's cheek to acquire skin conductance levels. This simultaneous acquisition of multimodal signals overcomes the problem of traditional single physiological signals being susceptible to interference or failing in a VR environment, providing a more comprehensive and robust data foundation for subsequent emotion recognition.

[0141] The acquired raw analog physiological signals are transmitted to an edge computing preprocessing unit embedded within the VR headset. An analog-to-digital converter first converts these analog signals into 24-bit high-precision digital signals. Subsequently, a digital signal processor (DSP) processes the digital signals. This processor is equipped with a bandpass filter bank to filter out noise outside a specific frequency range; it also features a motion artifact adaptive filter. This filter uses acceleration data acquired by the eye-tracking unit as a reference signal and employs an adaptive least mean square algorithm to filter out motion interference components in EEG and pulse wave signals in real time, effectively solving the problem of low recognition accuracy caused by motion artifacts in traditional physiological signals. Next, a sliding window slicer slices the filtered signals with a 3-second window length and a 0.5-second sliding step, generating multi-channel time-series data. Adjacent windows have a 75% overlap, and timestamps are aligned for each modality signal during sample generation to ensure temporal consistency of the multimodal data. These preprocessing steps are completed at the edge, reducing data transmission latency and improving data quality.

[0142] The preprocessed multi-channel time-series data is input into a multimodal fusion neural network model. This model first performs preliminary processing on the data from each modality through a spatial feature extraction layer. The spatial feature extraction layer consists of four parallel, structurally independent one-dimensional convolutional neural network branches, corresponding to the four modalities: EEG, eye movement, pulse wave, and electrodermal signaling (EDS). For example, the EEG signal branch has a 3×1 kernel size, a stride of 1, and output channels of 32, 64, and 128 respectively; the eye movement signal branch has a 5×1 kernel size, a stride of 1, and output channels of 16, 32, and 64 respectively; the pulse wave signal branch has a 4×1 kernel size, a stride of 1, and output channels of 16, 32, and 64 respectively; and the EDS signal branch has an 8×1 kernel size, a stride of 1, and output channels of 8, 16, and 32 respectively. The pooling layers for each branch all use 2×1 max pooling. Each branch extracts local spatial features of each modality through at least three convolutional blocks (consisting of one-dimensional convolutional layers, batch normalization layers, ReLU activation layers, and max pooling layers stacked together).

[0143] Subsequently, these spatial feature sequences are fed into a temporal feature extraction layer, which consists of a bidirectional long short-term memory network layer containing 128 hidden units. Its input dimension is the sum of the dimensions of the feature vectors output from each modality branch, and its output dimension is 256. This layer captures the forward and backward contextual information in each modality feature sequence, extracting temporally dependent feature vectors.

[0144] Next, the cross-modal attention fusion layer receives these temporal feature vectors. This layer includes a multi-head attention mechanism layer and a feature concatenation layer. The multi-head attention mechanism layer uses four attention heads, each with a dimension of 64. It takes the temporal feature vectors as query, key, and value inputs, and calculates the weight coefficients of each modality feature relative to the emotion recognition task through scaled dot product attention. These weight coefficients are updated in real time during the model inference phase. The feature concatenation layer concatenates the weighted features from each modality to form a fused feature vector. This fusion method effectively solves the problem of traditional methods lacking effective fusion of temporal and spatial features, achieving deep integration of multimodal information and improving the robustness of emotion recognition.

[0145] Finally, the classification output layer receives the fused feature vector. This layer consists of at least two fully connected layers and a Softmax output layer, mapping the fused feature vector to a probability distribution over a preset emotion category. The preset emotion categories include four quadrants: high arousal-high pleasure, high arousal-low pleasure, low arousal-high pleasure, and low arousal-low pleasure, as well as a neutral emotion category, for a total of five output categories.

[0146] The emotion state analysis unit receives the probability distribution generated by the classification output layer and outputs discrete emotion labels (e.g., "high arousal - low pleasure") and continuous pleasure-arousal coordinate values. The real-time rendering engine interface dynamically adjusts the presentation parameters of the virtual reality scene based on these emotion labels or coordinate values. For example, when the system detects that user A is in a "high arousal - low pleasure" state, the VR content adaptive adjustment module may, through the real-time rendering engine interface, reduce the tone mapping brightness of the virtual scene, adjust the ambient sound parameters to soothing music, or reduce the difficulty coefficient of the current interactive task to guide the user's emotion towards a "high pleasure" or "neutral" state. This dynamic, real-time content adjustment allows the VR experience to be personalized according to user A's real-time emotional state, significantly improving immersion and user experience, and overcoming the problem of insufficient immersion caused by the inability of traditional VR interaction to perceive user emotions.

[0147] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. An emotion recognition method based on a multimodal fusion neural network, characterized in that, include: Step S1: Acquire multimodal physiological signal data, including electroencephalogram (EEG) signals, eye movement data, heart rate variability signals, and skin conductance response signals; Step S2: The acquired raw signal is filtered, sliced, and normalized by the edge computing preprocessing unit to generate multi-channel time series data; Step S3: Input the multi-channel time-series data into a multimodal fusion neural network model. The multimodal fusion neural network model includes a spatial feature extraction layer, a temporal feature extraction layer, a cross-modal attention fusion layer, and a classification output layer, which are used to extract deep features of multimodal signals and identify user emotional states. The spatial feature extraction layer uses a graph convolutional network or a convolutional neural network to encode the independent spatial features of each modality; the temporal feature extraction layer uses a long short-term memory network or a gated recurrent unit to model the temporal dependence; and the cross-modal attention fusion layer uses a multi-head attention mechanism to weightedly fuse features from different modalities to generate a cross-modal joint representation. The classification output layer outputs the probability distribution of emotion categories based on the cross-modal joint representation, which serves as the emotion recognition result.

2. The emotion recognition method based on a multimodal fusion neural network according to claim 1, characterized in that, include: The VR wearable device body includes a head-mounted display shell, a flexible pad that fits the face, and a head-mounted fixing strap; The multimodal physiological signal data is integrated into the VR wearable device body and is used to synchronously collect the user's electroencephalogram (EEG) signals, eye movement data, heart rate variability signals, and skin conductance response signals; the VR content adaptive adjustment module is used to dynamically adjust the presentation parameters of the virtual reality scene according to the identified emotional state.

3. The emotion recognition method based on a multimodal fusion neural network according to claim 2, characterized in that, The multimodal physiological signal data specifically includes: At least four dry electrode EEG acquisition units are deployed at the corresponding Fp1 and Fp2 locations in the prefrontal lobe and T3 and T4 locations in the temporal lobe, respectively, for acquiring EEG signals; The binocular optical eye-tracking unit, consisting of an infrared light source and a high-speed camera, is deployed inside the head-mounted display at the position below the eye socket to collect changes in pupil diameter, blink frequency, and gaze trajectory. The photoelectric volumetric sensor consists of two dry electrodes deployed at the corresponding temple position to collect photoplethysmography (PPG) signals. The skin conductance sensor, consisting of two gold-plated dry electrodes, is deployed on one side of the cheek corresponding to the flexible pad to collect skin conductance levels.

4. The emotion recognition method based on a multimodal fusion neural network according to claim 2, characterized in that, The edge computing preprocessing unit is embedded inside the VR headset and specifically includes: Analog-to-digital converters are used to convert analog physiological signals into 24-bit high-precision digital signals; The digital signal processor is equipped with a bandpass filter bank, a motion artifact adaptive filter, and a sliding window slicer to generate time-aligned multi-channel time-series samples.

5. The emotion recognition method based on a multimodal fusion neural network according to claim 1, characterized in that, The multimodal fusion neural network model consists of the following structure: The spatial feature extraction layer consists of four parallel, structurally independent one-dimensional convolutional neural network branches, corresponding to four modalities: electroencephalography (EEG), eye movement, pulse wave, and skin conductance. Each branch includes at least three convolutional blocks, and each convolutional block is composed of a stack of one-dimensional convolutional layers, batch normalization layers, ReLU activation layers, and max pooling layers. The temporal feature extraction layer is composed of a bidirectional long short-term memory network layer. Its input is the four modal feature sequences output by the spatial feature extraction layer, and its output is a temporal feature vector with forward and backward context information. The bidirectional long short-term memory network layer contains 128 hidden units. Its input dimension is the sum of the dimensions of the feature vectors output by each modal branch, and its output dimension is 256. The cross-modal attention fusion layer includes a multi-head attention mechanism layer and a feature concatenation layer. The multi-head attention mechanism layer uses the temporal feature vector as the query, key, and value input to calculate the attention weight distribution of each modality. The feature concatenation layer concatenates the weighted features of each modality to form a fused feature vector. The multi-head attention mechanism layer uses four attention heads, each with a dimension of 64, and calculates the weight coefficients of each modality feature relative to the emotion recognition task by scaling dot product attention. The weight coefficients are updated in real time during the model inference stage. The classification output layer includes at least two fully connected layers and one Softmax output layer, which is used to map the fused feature vector to a probability distribution on a preset emotion category. The preset emotion category includes four quadrant categories: high arousal-high pleasure, high arousal-low pleasure, low arousal-high pleasure, and low arousal-low pleasure, as well as a neutral emotion category, for a total of five output categories.

6. The emotion recognition method based on a multimodal fusion neural network according to claim 1, characterized in that, An emotion state analysis unit is used to receive the probability distribution and output discrete emotion labels and continuous pleasure-arousal coordinate values; The real-time rendering engine interface is used to dynamically adjust the tone mapping, environmental sound effect parameters, narrative branch weights, or interaction difficulty coefficients of the virtual scene based on the emotion tags or coordinate values.

7. The emotion recognition method based on a multimodal fusion neural network according to claim 4, characterized in that, The motion artifact adaptive filter is equipped with a real-time motion compensation algorithm based on eye-tracking signals. The algorithm uses acceleration data collected by the eye-tracking unit as a reference signal and filters out motion interference components in EEG signals and pulse wave signals through an adaptive least mean square algorithm.

8. The emotion recognition method based on a multimodal fusion neural network according to claim 4, characterized in that, The sliding window slicer is configured with a window length of 3 seconds, a sliding step size of 0.5 seconds, and a 75% overlap rate between adjacent windows. Furthermore, the slicer timestamps each modal signal when generating samples to ensure the temporal consistency of multimodal data.

9. The emotion recognition method based on a multimodal fusion neural network according to claim 5, characterized in that, In the four parallel one-dimensional convolutional neural network branches: The convolution kernel size of the EEG signal branch is 3×1, the stride is 1, and the number of output channels is 32, 64, and 128 respectively. The convolution kernel size of the eye-tracking signal branch is 5×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The convolution kernel size of the pulse wave signal branch is 4×1, the stride is 1, and the number of output channels is 16, 32, and 64 respectively. The kernel size of the electrodermal signal branch is 8×1, the stride is 1, and the number of output channels is 8, 16, and 32 respectively. Each branch's pooling layer uses max pooling with a size of 2×1.

10. The emotion recognition method based on a multimodal fusion neural network according to claim 2, characterized in that: The head-mounted display shell has a first connecting strap and a second connecting strap connected to its left and right ends, respectively. The ends of the first and second connecting straps are connected via a connecting box. The connecting box has openings at both the front and rear ends, and drive motors are embedded in the left and right ends of the openings. The output ends of the drive motors are connected to drive gears. The outer ends of the first and second connecting straps are provided with racks that mesh with the drive gears. The ends of the first and second connecting straps are provided with anti-detachment blocks. An auxiliary fixing strap is provided between the middle of the head-mounted display shell and the connecting box. The auxiliary fixing strap is provided with telescopic adjustment buckles. Multiple connecting wires are connected to the head-mounted display shell, and electrode plates are connected to the ends of the connecting wires.

Images