A Multimodal Emotion Recognition Method and System Based on VR Intervention

By employing a multimodal synchronous acquisition and adaptive weighted fusion method, the problem of device motion disturbance and modal response differences in virtual reality interaction was solved, achieving highly accurate and stable emotional state recognition.

CN123120322APending Publication Date: 2026-11-17NINGBO XINYI DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611474730.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-21
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing single-modal emotion recognition methods based on virtual reality interaction technology struggle to accurately reflect user emotional changes when faced with device motion disturbances and modal response differences, resulting in insufficient recognition accuracy.

Method used

By employing multimodal synchronous acquisition, dynamic correction of visual signals, extraction of emotional response features, and cross-modal emotional response analysis, a motion perturbation index and visual credibility are established. By combining the temporal correlation between visual and physiological signals, adaptive weighted fusion and time compensation are performed to achieve cross-modal emotional response difference analysis.

Benefits of technology

It improves the accuracy and stability of emotion state recognition in virtual reality environments, makes full use of the complementary relationship between visual and physiological information, and reduces the impact of device motion on the extraction of visual emotion features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123120322A_ABST
    Figure CN123120322A_ABST
Patent Text Reader

Abstract

This invention discloses a multimodal emotion recognition method and system based on VR intervention, belonging to the field of virtual reality interaction technology. The method includes multimodal synchronous acquisition, dynamic correction of visual signals, extraction of emotional response features, and cross-modal emotion response analysis. This invention establishes a motion perturbation index and constructs visual credibility by combining facial image quality features. Based on the visual credibility, it dynamically weights and fuses current visual features with historical visual features to achieve adaptive correction of facial visual signals under motion perturbation conditions. By analyzing the response correlation between different modal emotional responses, it obtains a cross-modal emotion response difference index and combines the differences in response time, amplitude of change, and trend of change between different modalities to identify emotional states. This fully utilizes the complementary relationship between visual and physiological information, improving the accuracy of user emotional state recognition in VR intervention scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of virtual reality interaction technology, specifically referring to a multimodal emotion recognition method and system based on VR intervention. Background Technology

[0002] With the development of virtual reality (VR) technology, emotion intervention, psychological training, and user status assessment based on immersive environments have gradually attracted attention. During virtual reality interaction, changes in a user's facial expressions and physiological states can reflect their emotional responses. Therefore, achieving emotion recognition by integrating visual and physiological information has become an important technological direction for improving intelligent interaction capabilities.

[0003] Existing emotion recognition methods based on virtual reality scenarios typically rely on single visual information or single physiological signals for emotion judgment. Facial image-based emotion recognition methods infer emotions by analyzing changes in the user's facial expressions; however, limitations imposed by the way virtual reality devices are worn and the near-eye acquisition environment can lead to incomplete facial region capture and insufficient representation of visual features. While physiological signal-based recognition methods can reflect changes in the user's autonomic nervous activity, there is a lack of direct correlation between physiological signals and external emotional expressions, making it difficult to accurately represent the user's emotional state independently. Therefore, single-modal information cannot comprehensively and accurately reflect the dynamic emotional changes during virtual reality intervention.

[0004] Meanwhile, during actual virtual reality interaction, users exhibit natural movement behaviors such as head rotation, movement, and posture changes. The motion disturbances of the head-mounted display device further affect the near-eye visual acquisition results, causing problems such as blurred facial images and regional shifts, reducing the reliability of visual emotional features. Furthermore, visual emotional responses and physiological emotional responses are influenced by human perception and neural regulation mechanisms, and the two may differ in response time, amplitude of change, and trend. Existing multimodal emotion recognition methods mostly employ feature splicing or simple fusion, lacking analytical mechanisms for the temporal relationships of responses across different modalities and the degree of cross-modal response mismatch. This makes it difficult to effectively utilize complementary information between different modalities, resulting in insufficient accuracy in emotion recognition under complex virtual reality interaction environments.

[0005] Therefore, there is an urgent need for a method that can dynamically correct visual signals in response to motion disturbances during virtual reality interaction, and combine the temporal correlation between visual and physiological emotional responses and cross-modal variation characteristics to achieve highly reliable multi-emotional state recognition. Summary of the Invention

[0006] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a multimodal emotion recognition method and system based on VR intervention. The method includes the following steps:

[0007] Step S1: Multimodal synchronous acquisition;

[0008] Step S2: Dynamic correction of visual signals;

[0009] Step S3: Extraction of emotional response features;

[0010] Step S4: Cross-modal emotion response analysis.

[0011] Further, in step S1, the multimodal synchronous acquisition is used to acquire user visual response information, physiological response information, and device motion state information during VR intervention. Specifically, based on the real-time interaction state after the user enters the VR intervention scene, facial image data, physiological response data, and head-mounted display motion state data in the virtual reality head-mounted display device are acquired, and the different modal data are timestamped and correlated based on the virtual reality rendering synchronization clock to obtain multimodal synchronous data.

[0012] Further, in step S2, the dynamic correction of the visual signal specifically involves analyzing the user's motion disturbance state in the virtual reality environment based on multimodal synchronization data to obtain a motion disturbance index, evaluating the credibility of facial image data to obtain visual credibility, and adaptively weighting and fusing the current visual features and historical visual features according to the visual credibility to obtain disturbance-compensated visual feature data, including the following steps:

[0013] Step S21 Head-mounted display motion disturbance state modeling, specifically, normalize the head-mounted display motion state data in the multimodal synchronization data to obtain head-mounted display motion normalization features, and weight and fuse each feature dimension in the head-mounted display motion normalization features to obtain a motion disturbance index, which is used to characterize the degree of disturbance of the head-mounted display motion to the facial visual signal during the current virtual reality interaction process.

[0014] Step S22 Facial visual quality credibility evaluation specifically involves extracting facial visual quality features from facial image data in multimodal synchronization data, and constructing visual credibility based on facial visual quality features and motion perturbation index to characterize the reliability of current facial visual information under virtual reality motion perturbation conditions.

[0015] Step S23, dynamic feature correction based on visual credibility, specifically involves extracting visual features from facial image data in multimodal synchronization data, and combining visual credibility to adaptively weight and fuse the current visual features and historical visual features to obtain perturbation-compensated visual feature data.

[0016] Furthermore, in step S3, the extraction of emotional response features specifically involves obtaining visual emotional response alignment features and physiological emotional response alignment features based on perturbation-compensated visual feature data and multimodal synchronization data, through facial feature change analysis, physiological signal feature parsing, and timestamp matching resampling processing.

[0017] Further, in step S4, the cross-modal emotion response analysis specifically involves analyzing the evolution process of cross-modal emotion response differences based on the response correlation between visual emotion response alignment features and physiological emotion response alignment features, obtaining a cross-modal emotion response difference index, constructing multimodal emotion state features, performing emotion state recognition, and obtaining the user's current emotion state category, including the following steps:

[0018] Step S41 Cross-modal response association analysis specifically involves using a sliding time window to perform response association analysis on visual emotion response alignment features and physiological emotion response alignment features, calculating the degree of response association between the two types of emotion response alignment features under different time offsets, determining the time offset corresponding to the maximum degree of response association as the optimal time offset between visual emotion response and physiological emotion response, and obtaining the corresponding maximum degree of response association.

[0019] Step S42 Cross-modal emotion response difference evolution analysis, specifically, based on the optimal time offset, performs time compensation on the physiological emotion response alignment features, calculates the response amplitude difference between the visual emotion response alignment features and the time-compensated physiological emotion response alignment features, obtains the emotion response difference change trend based on the change of response amplitude difference in the continuous time window, and obtains the emotion response difference persistence state features by performing persistence analysis on the response amplitude difference.

[0020] Step S43 Calculation of cross-modal emotion response difference index: Specifically, the cross-modal emotion response difference index is obtained by fusing the maximum response correlation degree, optimal time offset, response amplitude difference, emotion response difference persistence state characteristics and emotion response difference change trend. This index is used to characterize the degree of inconsistency between visual emotion response and physiological emotion response.

[0021] Step S44, emotion state recognition, specifically involves constructing multimodal emotion state features based on visual emotion response alignment features, physiological emotion response alignment features, and cross-modal emotion response difference index. These multimodal emotion state features are then input into the emotion state recognition model for classification processing, and the user's current emotion state category is output.

[0022] This invention provides a multimodal emotion recognition system based on VR intervention, including a multimodal synchronous acquisition module, a visual signal dynamic correction module, an emotion response feature extraction module, and a cross-modal emotion response analysis module;

[0023] The multimodal synchronous acquisition module is used for multimodal synchronous acquisition, to obtain multimodal synchronous data, and to send the multimodal synchronous data to the visual signal dynamic correction module and the emotion response feature extraction module.

[0024] The visual signal dynamic correction module is used to dynamically correct the visual signal, obtain perturbation-compensated visual feature data, and send the perturbation-compensated visual feature data to the emotion response feature extraction module.

[0025] The emotion response feature extraction module is used to extract emotion response features, obtain visual emotion response alignment features and physiological emotion response alignment features, and send the visual emotion response alignment features and physiological emotion response alignment features to the cross-modal emotion response analysis module.

[0026] The cross-modal emotion response analysis module is used for cross-modal emotion response analysis to obtain the user's current emotion state category.

[0027] The beneficial effects achieved by the present invention using the above solution are as follows:

[0028] By establishing a motion perturbation index and constructing visual credibility based on facial image quality features, and dynamically weighting and fusing current and historical visual features according to the visual credibility, adaptive correction of facial visual signals under motion perturbation conditions is achieved. This reduces the impact of factors such as user head movement and posture changes on the visual emotion feature extraction process, improving the stability and reliability of visual emotion feature extraction. By analyzing the response correlation between different modal emotional responses, a cross-modal emotional response difference index is obtained. Emotional state recognition is then performed by combining the differences in response time, change amplitude, and change trend between different modalities. This fully utilizes the complementary relationship between visual and physiological information, improving the accuracy of user emotional state recognition in VR intervention scenarios. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a VR-based multimodal emotion recognition method provided by this invention;

[0030] Figure 2 This is a schematic diagram of a multimodal emotion recognition system based on VR intervention provided by the present invention;

[0031] Figure 3 This is a flowchart illustrating step S2;

[0032] Figure 4 This is a flowchart illustrating step S4.

[0033] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0034] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0035] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] Example 1, see Figure 1 This invention provides a multimodal emotion recognition method and system based on VR intervention. The method includes the following steps:

[0037] Step S1: Multimodal synchronous acquisition;

[0038] Step S2: Dynamic correction of visual signals;

[0039] Step S3: Extraction of emotional response features;

[0040] Step S4: Cross-modal emotion response analysis.

[0041] Example 2, see Figure 1 This embodiment is based on the above embodiment. In step S1, the multimodal synchronous acquisition is used to acquire user visual response information, physiological response information and device motion state information during virtual reality intervention. Specifically, based on the real-time interaction state after the user enters the virtual reality intervention scene, facial image data, physiological response data and head display motion state data in the virtual reality head display device are collected, and the different modal data are timestamped and associated based on the virtual reality rendering synchronization clock to obtain multimodal synchronous data under a unified time reference.

[0042] The acquisition of facial image data specifically involves continuously acquiring facial image data containing information on changes in the eyes, eyebrows, and facial texture through a near-eye vision acquisition unit located inside the virtual reality headset. This data is used to characterize the user's facial response state during virtual reality interaction.

[0043] The collection of the physiological response data specifically involves acquiring the user's pulse wave signal through a physiological signal acquisition unit installed in a virtual reality device or an external wearable device, and recording the pulse wave signal over time to obtain physiological response data reflecting changes in the user's autonomic nervous activity.

[0044] The acquisition of the device motion state data specifically involves obtaining motion state information of the head-mounted display in three-dimensional space through the inertial measurement unit in the virtual reality head-mounted display device, thereby obtaining head-mounted display motion state data. The head-mounted display motion state data includes head-mounted display angular velocity, linear velocity, and attitude change, which are used to describe the user's head motion state.

[0045] The timestamp association processing specifically involves using the virtual reality rendering synchronization clock as a unified time reference to add corresponding timestamps to the facial image data, physiological response data, and head-mounted display motion state data, and converting data with different sampling frequencies to a unified time axis according to the timestamp mapping relationship to form multimodal synchronized data.

[0046] Example 3, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S2, the dynamic correction of the visual signal is used to reduce the impact of user motion disturbance on facial visual emotion feature extraction during virtual reality intervention. Specifically, based on multimodal synchronous data, the motion disturbance state of the user in the virtual reality environment is analyzed to obtain a motion disturbance index, and the credibility of the facial image data is evaluated to obtain visual credibility. Based on the visual credibility, the current visual features and historical visual features are adaptively weighted and fused to obtain disturbance-compensated visual feature data, including the following steps:

[0047] Step S21 Head-mounted display motion disturbance state modeling, specifically, normalize the head-mounted display motion state data in the multimodal synchronization data to obtain head-mounted display motion normalization features, and weight and fuse each feature dimension in the head-mounted display motion normalization features to obtain a motion disturbance index, which is used to characterize the degree of disturbance of the head-mounted display motion to the facial visual signal during the current virtual reality interaction process.

[0048] In specific implementation, the head-mounted display motion normalization feature consists of multiple normalized motion feature dimensions, including angular velocity normalization feature, linear velocity normalization feature, and posture change normalization feature.

[0049] The formula for calculating the motion disturbance index is as follows:

[0050] ;

[0051] In the formula, R m (t) represents the current motion disturbance index, t represents the time index, a1, a2, a3 represent the weight parameters corresponding to different motion characteristics, x1(t) represents the current angular velocity normalization feature, x2(t) represents the current linear velocity normalization feature, and x3(t) represents the attitude change normalization feature.

[0052] Step S22 Facial visual quality credibility evaluation specifically involves extracting facial visual quality features from facial image data in multimodal synchronization data, and constructing visual credibility based on facial visual quality features and motion perturbation index to characterize the reliability of current facial visual information under virtual reality motion perturbation conditions.

[0053] In specific implementation, the facial visual quality features include image sharpness features and facial region stability features;

[0054] The formula for calculating the image sharpness feature is as follows:

[0055] ;

[0056] In the formula, Q b (t) represents the image sharpness feature, and Var(·) represents the variance operation function. Indicates gradient changes in a facial image;

[0057] The formula for calculating the stable features of the facial region is as follows:

[0058] ;

[0059] In the formula, Q p (t) represents stable features of the facial region, d f (t) represents the current offset distance of the facial key point, d max Indicates the maximum permissible offset distance for facial key points;

[0060] The formula for calculating visual credibility is:

[0061] ;

[0062] In the formula, C v (t) represents visual credibility. denoted by , b1, b2, and b3 represent the weight parameters corresponding to image sharpness features, facial region stability features, and motion perturbation index, respectively.

[0063] Step S23 is dynamic feature correction based on visual credibility. Specifically, it involves extracting visual features from facial image data in multimodal synchronization data, and combining visual credibility to adaptively weight and fuse the current visual features and historical visual features to obtain perturbation-compensated visual feature data.

[0064] In specific implementation, the visual features are obtained by visual feature encoding of facial image data. Specifically, the facial image is input into a visual feature extraction model, and the texture information, morphological change information, and expression change information of the facial region are encoded to obtain the current visual features. The visual feature extraction model can be implemented using a convolutional neural network, a visual Transformer model, or other models with image feature encoding capabilities.

[0065] The calculation formula for the adaptive weighted fusion is as follows:

[0066] ;

[0067] In the formula, F represents the visual features compensated for perturbation. v (t) represents the current visual feature. Indicates historical visual characteristics, This indicates the visual feature update cycle, used to determine the time interval between the current visual feature and historical visual features.

[0068] By performing the above operations, this solution establishes a motion perturbation index and constructs visual credibility by combining facial image quality features. Based on the visual credibility, it dynamically weights and fuses current visual features with historical visual features, thereby achieving adaptive correction of facial visual signals under motion perturbation conditions. This reduces the impact of factors such as user head movement and posture changes on the visual emotion feature extraction process, and improves the stability and reliability of visual emotion feature extraction.

[0069] Example 4, see Figure 1 This embodiment is based on the above embodiment. In step S3, the emotional response feature extraction is specifically based on perturbation-compensated visual feature data and multimodal synchronization data. Through facial feature change analysis, physiological signal feature parsing and timestamp matching resampling processing, visual emotional response alignment features and physiological emotional response alignment features are obtained.

[0070] In specific implementation, the facial feature change analysis specifically involves using methods such as facial key point localization, facial region change detection, and local feature extraction on the perturbation-compensated visual feature data to analyze the positional changes, texture changes, and motion intensity information of the eye region, eyebrow region, and local facial regions over time, thereby obtaining visual emotional response features that characterize changes in the user's facial state.

[0071] The physiological signal feature analysis specifically involves preprocessing and analyzing the physiological response data in the multimodal synchronous data, and obtaining physiological emotional response features that reflect changes in the user's physiological state by analyzing changes in pulse wave cycle and autonomic nervous activity response information.

[0072] The timestamp matching and resampling process specifically involves performing timestamp matching on the visual emotion response features and physiological emotion response features based on the virtual reality rendering synchronization clock in step S1, performing time resampling according to the sampling frequency differences of different feature sequences, and mapping the visual emotion response features and physiological emotion response features to the same time axis to obtain visual emotion response alignment features and physiological emotion response alignment features.

[0073] Example 5, see Figure 1 and Figure 4 This embodiment is based on the above embodiment. In step S4, the cross-modal emotion response analysis specifically involves analyzing the evolution process of cross-modal emotion response differences based on the response correlation between visual emotion response alignment features and physiological emotion response alignment features, obtaining a cross-modal emotion response difference index, constructing multimodal emotion state features, performing emotion state recognition, and obtaining the user's current emotion state category. This includes the following steps:

[0074] Step S41 Cross-modal response association analysis specifically involves using a sliding time window to perform response association analysis on visual emotion response alignment features and physiological emotion response alignment features, calculating the degree of response association between the two types of emotion response alignment features under different time offsets, determining the time offset corresponding to the maximum degree of response association as the optimal time offset between visual emotion response and physiological emotion response, and obtaining the corresponding maximum degree of response association.

[0075] In practical implementation, the formula for calculating the degree of response correlation is as follows:

[0076] ;

[0077] In the formula, Indicates the time offset as The degree of correlation between the two types of sentiment response alignment features corresponding to the current time window, where k represents the time window index. This represents the time offset, where i represents the index of the time point, and N represents the number of time points within the current time window. This represents the visual emotion response alignment feature corresponding to the i-th time point. This represents the mean of the visual emotion response alignment features within the current time window. Indicates the first Alignment characteristics of physiological and emotional responses at specific time points This represents the mean of the alignment features of physiological emotional responses within the current time window. This represents the standard deviation of the visual emotion response alignment features within the current time window. This represents the standard deviation of the alignment characteristics of physiological and emotional responses within the current time window.

[0078] Step S42 Cross-modal emotion response difference evolution analysis, specifically, based on the optimal time offset, performs time compensation on the physiological emotion response alignment features, calculates the response amplitude difference between the visual emotion response alignment features and the time-compensated physiological emotion response alignment features, obtains the emotion response difference change trend based on the change of response amplitude difference in the continuous time window, and obtains the emotion response difference persistence state features by performing persistence analysis on the response amplitude difference.

[0079] In practice, the time compensation is used to correct the dynamic response delay between visual and physiological emotional responses caused by differences in human emotional response mechanisms.

[0080] The formula for calculating the difference in response amplitude is:

[0081] ;

[0082] In the formula, D k This represents the difference in response amplitude between the visual emotion response alignment feature corresponding to the current time window and the time-compensated physiological emotion response alignment feature. This indicates the alignment characteristics of physiological and emotional responses after time compensation. This represents the optimal time offset corresponding to the current time window;

[0083] The calculation formula for the persistence analysis is as follows:

[0084] ;

[0085] In the formula, P k This indicates the persistent state characteristics of the emotional response differences corresponding to the current time window, L represents the number of historical time windows involved in the persistent analysis, and j represents the historical time window backtracking sequence number;

[0086] Step S43 Calculation of cross-modal emotion response difference index: Specifically, the cross-modal emotion response difference index is obtained by fusing the maximum response correlation degree, optimal time offset, response amplitude difference, emotion response difference persistence state characteristics and emotion response difference change trend. This index is used to characterize the degree of inconsistency between visual emotion response and physiological emotion response.

[0087] In practical implementation, the calculation formula for the cross-modal emotion response difference index is as follows:

[0088] ;

[0089] In the formula, E k This represents the cross-modal sentiment response difference index corresponding to the current time window. This represents the maximum correlation between the two types of sentiment response alignment features corresponding to the k-th time window. G represents the preset time offset threshold. k This indicates the changing trend of emotional response differences within the current time window. , , , , These represent the weighting coefficients corresponding to the maximum response correlation, optimal time offset, response amplitude difference, emotional response difference persistence characteristics, and emotional response difference change trend, respectively.

[0090] Step S44, emotion state recognition, specifically involves constructing multimodal emotion state features based on visual emotion response alignment features, physiological emotion response alignment features, and cross-modal emotion response difference index. The multimodal emotion state features are then input into the emotion state recognition model for classification processing, and the user's current emotion state category is output.

[0091] In specific implementation, the emotion state recognition model can be implemented using a classification model based on deep neural networks, including but not limited to multilayer perceptron networks, long short-term memory networks, gated recurrent unit networks, or Transformer networks.

[0092] The emotional state categories may include stable response state, positive response state, fluctuating response state, and negative response state.

[0093] By performing the above operations, this solution analyzes the response correlation between different modal emotional responses, obtains the cross-modal emotional response difference index, and combines the differences in response time, change magnitude, and change trend between different modalities to identify emotional states. This fully utilizes the complementary relationship between visual and physiological information, and improves the accuracy of user emotional state identification in VR intervention scenarios.

[0094] Example 6, see Figure 2 Based on the above embodiments, this embodiment provides a multimodal emotion recognition system based on VR intervention, including a multimodal synchronous acquisition module, a visual signal dynamic correction module, an emotion response feature extraction module, and a cross-modal emotion response analysis module;

[0095] The multimodal synchronous acquisition module is used for multimodal synchronous acquisition, to obtain multimodal synchronous data, and to send the multimodal synchronous data to the visual signal dynamic correction module and the emotion response feature extraction module.

[0096] The visual signal dynamic correction module is used to dynamically correct the visual signal, obtain perturbation-compensated visual feature data, and send the perturbation-compensated visual feature data to the emotion response feature extraction module.

[0097] The emotion response feature extraction module is used to extract emotion response features, obtain visual emotion response alignment features and physiological emotion response alignment features, and send the visual emotion response alignment features and physiological emotion response alignment features to the cross-modal emotion response analysis module.

[0098] The cross-modal emotion response analysis module is used for cross-modal emotion response analysis to obtain the user's current emotion state category.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0100] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0101] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A multimodal emotion recognition method based on VR intervention, characterized in that: The method includes the following steps: Step S1: Multimodal synchronous acquisition to obtain multimodal synchronous data; Step S2, visual signal dynamic correction, specifically involves analyzing the user's motion disturbance state in the virtual reality environment based on multimodal synchronization data to obtain a motion disturbance index, evaluating the credibility of facial image data to obtain visual credibility, and adaptively weighting and fusing the current visual features and historical visual features based on the visual credibility to obtain disturbance compensation visual feature data. The sub-steps of step S2 include step S21 modeling the motion disturbance state of the head display, step S22 evaluating the credibility of facial visual quality, and step S23 correcting the dynamic features based on visual credibility. Step S3 extracts emotional response features, resulting in visual emotional response alignment features and physiological emotional response alignment features; Step S4, cross-modal emotion response analysis, specifically involves analyzing the evolution process of cross-modal emotion response differences based on the response correlation between visual emotion response alignment features and physiological emotion response alignment features, obtaining a cross-modal emotion response difference index, constructing multimodal emotion state features, performing emotion state recognition, and obtaining the user's current emotion state category. The sub-steps of step S4 include step S41 cross-modal response correlation analysis, step S42 cross-modal emotion response difference evolution analysis, step S43 cross-modal emotion response difference index calculation, and step S44 emotion state identification.

2. The multimodal emotion recognition method based on VR intervention according to claim 1, characterized in that: Step S21 Modeling the motion disturbance state of the head-mounted display (HMD) involves normalizing the HMD motion state data in the multimodal synchronization data to obtain the HMD motion normalization features, and then weighting and fusing the feature dimensions in the HMD motion normalization features to obtain the motion disturbance index, which is used to characterize the degree of disturbance to the facial visual signals caused by the HMD motion during the current virtual reality interaction process.

3. The multimodal emotion recognition method based on VR intervention according to claim 2, characterized in that: Step S22, Facial Visual Quality Credibility Evaluation, specifically involves extracting facial visual quality features from facial image data in multimodal synchronization data, and constructing visual credibility based on facial visual quality features and motion perturbation index to characterize the reliability of current facial visual information under virtual reality motion perturbation conditions.

4. The multimodal emotion recognition method based on VR intervention according to claim 3, characterized in that: Step S23, dynamic feature correction based on visual credibility, specifically involves extracting visual features from facial image data in multimodal synchronization data, and combining visual credibility to adaptively weight and fuse the current visual features and historical visual features to obtain perturbation-compensated visual feature data.

5. The multimodal emotion recognition method based on VR intervention according to claim 4, characterized in that: Step S41 Cross-modal response association analysis specifically involves using a sliding time window to perform response association analysis on visual emotion response alignment features and physiological emotion response alignment features, calculating the degree of response association between the two types of emotion response alignment features under different time offsets, determining the time offset corresponding to the maximum degree of response association as the optimal time offset between visual emotion response and physiological emotion response, and obtaining the corresponding maximum degree of response association. Step S42, cross-modal emotion response difference evolution analysis, specifically involves performing time compensation on the physiological emotion response alignment features based on the optimal time offset, calculating the response amplitude difference between the visual emotion response alignment features and the time-compensated physiological emotion response alignment features, obtaining the trend of emotion response difference changes based on the changes in response amplitude difference in a continuous time window, and obtaining the persistent state features of emotion response difference by performing persistence analysis on the response amplitude difference.

6. The multimodal emotion recognition method based on VR intervention according to claim 5, characterized in that: Step S43 Calculation of cross-modal emotion response difference index: Specifically, the cross-modal emotion response difference index is obtained by fusing the maximum response correlation degree, optimal time offset, response amplitude difference, emotion response difference persistence state characteristics and emotion response difference change trend. This index is used to characterize the degree of inconsistency between visual emotion response and physiological emotion response. Step S44, emotion state recognition, specifically involves constructing multimodal emotion state features based on visual emotion response alignment features, physiological emotion response alignment features, and cross-modal emotion response difference index. These multimodal emotion state features are then input into the emotion state recognition model for classification processing, and the user's current emotion state category is output.

7. The multimodal emotion recognition method based on VR intervention according to claim 6, characterized in that: In step S3, the emotional response feature extraction specifically involves obtaining visual emotional response alignment features and physiological emotional response alignment features based on perturbation-compensated visual feature data and multimodal synchronization data, through facial feature change analysis, physiological signal feature parsing, and timestamp matching resampling processing.

8. The multimodal emotion recognition method based on VR intervention according to claim 7, characterized in that: In step S1, the multimodal synchronous acquisition is used to acquire user visual response information, physiological response information, and device motion state information during VR intervention. Specifically, based on the real-time interaction state of the user after entering the VR intervention scene, facial image data, physiological response data, and head-mounted display motion state data in the virtual reality head-mounted display device are acquired, and the different modal data are timestamped and correlated based on the virtual reality rendering synchronization clock to obtain multimodal synchronous data.

9. A VR-based multimodal emotion recognition system, used to implement the VR-based multimodal emotion recognition method as described in any one of claims 1-7, characterized in that: It includes a multimodal synchronous acquisition module, a visual signal dynamic correction module, an emotion response feature extraction module, and a cross-modal emotion response analysis module; The multimodal synchronous acquisition module is used for multimodal synchronous acquisition, to obtain multimodal synchronous data, and to send the multimodal synchronous data to the visual signal dynamic correction module and the emotion response feature extraction module. The visual signal dynamic correction module is used to dynamically correct the visual signal, obtain perturbation-compensated visual feature data, and send the perturbation-compensated visual feature data to the emotion response feature extraction module. The emotion response feature extraction module is used to extract emotion response features, obtain visual emotion response alignment features and physiological emotion response alignment features, and send the visual emotion response alignment features and physiological emotion response alignment features to the cross-modal emotion response analysis module. The cross-modal emotion response analysis module is used for cross-modal emotion response analysis to obtain the user's current emotion state category.