Multi-modal fusion-based VR emotion recognition and response system and method
Through the multimodal fusion VR emotion recognition system, real-time analysis of user facial expressions, voice and biological signals, the problems of low emotional recognition accuracy and undynamic adjustment in the existing technology are solved, and more accurate emotional recognition and personalization are achieved VR experience enhances emotional resonance and cultural communication effects.
Patent Information
- Application Number
- CN202410975139.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-12-03
AI Technical Summary
While pursuing visual and auditory experiences, the existing VR system ignores the perception and response of the user's emotional state, resulting in low emotional recognition accuracy and inability to dynamically adjust content, weakening the personalization of emotional resonance and user experience.
Using a VR emotion recognition and response system based on multimodal fusion, the multimodal data is analyzed in real time through data collection, preprocessing, feature extraction and DCCA network module, and a unified emotional representation is output, and the VR content is intelligently adjusted according to the emotional recognition results. Provide a personalized experience.
It significantly improves the accuracy and robustness of emotional recognition, dynamically adjusts VR content, provides users with an immersive experience that is closer to personal emotional needs, and promotes emotional resonance and the dissemination of intangible cultural heritage.
Smart Images

Figure CN119066606A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of emotion recognition technology, and specifically relates to a VR emotion recognition and response system and method based on multimodal fusion. Background Art
[0002] With the rapid development of VR technology (virtual reality technology), people's demand for immersive experience is growing. VR technology is not limited to the fields of games and entertainment, and its application scope has expanded to many fields such as education, medical care, psychotherapy, and architectural design. In order to provide a more realistic and fascinating experience, existing VR systems usually rely on advanced graphics rendering technology, interactive design, and high-fidelity audio-visual effects. However, while pursuing the ultimate visual and auditory experience, these systems often ignore the perception and response of the user's emotional state, which has become one of the key factors limiting the further deepening of the VR experience.
[0003] Currently, VR systems provide immersive experiences mainly through the following methods:
[0004] Graphics rendering: Use high-performance graphics processors and advanced rendering algorithms to generate high-resolution, high-frame-rate virtual scenes to simulate the visual experience of the real world.
[0005] Interaction design: Through technologies such as handles, gesture recognition, and eye tracking, users can interact with the virtual environment naturally and intuitively.
[0006] Audio processing: Use stereo effects, surround sound effects and other technologies to create an auditory environment with a strong sense of space and enhance immersion. In the field of emotion recognition, researchers have developed single-modal emotion recognition technology based on facial expression recognition and voice emotion analysis. For example, by analyzing the changes in key points of the face, expressions such as smiles and frowns can be recognized, or by analyzing the tone, speed, volume and other characteristics of the voice, the speaker's emotional state can be judged. However, emotion recognition technology using a single modality can often only capture part of the emotional expression, while the expression of human emotions is multi-dimensional, including facial expressions, voice intonation, body posture, physiological signals, etc. Single-modal recognition is prone to missing important information, resulting in low accuracy and reliability of emotion recognition. In addition, existing VR systems lack the ability to dynamically adjust content according to user emotions, which means that no matter how the user's emotional state changes, the VR experience is fixed and cannot be adjusted according to the user's real-time emotional feedback, which not only weakens emotional resonance, but also reduces the personalization and immersion of the user experience.
[0007] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention
[0008] The purpose of the present invention is to provide a VR emotion recognition and response system and method based on multimodal fusion, which can solve the problems raised in the above background technology.
[0009] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:
[0010] The VR emotion recognition and response system based on multimodal fusion includes a data acquisition module, a data preprocessing module, a feature extraction module, a DCCA network module, an emotion recognition module and a response generation module. The data acquisition module is used to capture the user's body language feature signals, oral language feature signals and biometric feature signals in real time. The data preprocessing module is used to preprocess the data collected by the data acquisition module. The feature extraction module is used to extract the key features of the data processed by the data preprocessing module. The DCCA network module is used to learn the deep association between different modal data to ensure the consistency and integrity of emotion expression. The emotion recognition module analyzes multimodal data in real time based on the DCCA network module and outputs a unified emotion expression. The response generation module intelligently adjusts VR content based on the recognition results of the emotion recognition module to provide personalized experience.
[0011] In one or more embodiments of the present invention, the data acquisition module includes a facial expression image acquisition module, a voice and audio acquisition module and a biometric signal acquisition module. The facial expression image acquisition module is used to collect facial expressions, and uses the OpenCV library for facial detection, and combines FACS to annotate the expression state. The voice and audio acquisition module is integrated on the VR device to collect the voice of the VR device user, and uses the SoundFile library to read the audio, and uses the librosa library to extract audio features. The biometric signal acquisition module is integrated on the VR device, using Arduino as a data acquisition platform to store signal data in real time.
[0012] In one or more embodiments of the present invention, the data preprocessing module includes an image processing module, an audio processing module, and a signal processing module. The image processing module uses the conversion function in the Pillow library to convert the image into a grayscale image and scale the pixel value to the interval [0, 1]. The audio processing module uses the VAD function in the pyAudioAnalysis library to achieve audio noise reduction and segmentation. The signal processing module uses the Butterworth filter in the SciPy library to filter the signal and uses the Resample library to ensure that the signal has the same sampling rate.
[0013] In one or more embodiments of the present invention, the feature extraction module includes a facial feature extraction module, a voice intonation feature extraction module, and a physiological feature extraction module. The facial feature extraction module uses a ResNet or VGG16 model pre-trained under a TensorFlow or PyTorch framework to extract feature vectors. The voice intonation feature extraction module uses a Wav2Vec or DeepSpeech model to extract voice intonation features, and voice intonation features include pitch, rhythm, and intensity. The physiological feature extraction module extracts physiological indicators from biometric signals through wavelet transform or fast Fourier transform.
[0014] In one or more embodiments of the present invention, an emotion-driven content adaptation module is included to adjust the content in the VR device according to the user's emotional state to enhance the user experience.
[0015] In one or more embodiments of the present invention, the content adaptation module includes a state classification module, a label mapping module and a preset response module. The state classification module is used to classify the emotional state obtained after multimodal emotion fusion. The label mapping module maps the emotional state to the label according to the user's current emotional state. The preset response module establishes a content library containing a variety of emotion-oriented contents, presets response rules according to the emotion-oriented contents, and specifies how the system should adjust the content when a specific emotional state is detected.
[0016] In one or more embodiments of the present invention, the content adaptation module includes an adjustment module, which is used to monitor the user's emotional state and adjust the content according to the real-time changes in the user's emotional state. The adjustment module uses a deep learning model to continuously analyze the user's multimodal data, including facial expressions, voice and biological signals, to identify the emotional state in real time, set the emotional change threshold, and immediately trigger the content adjustment process when a significant change is detected. According to the emotional label, the system retrieves the VR scene, music and interactive elements that match it, so as to achieve an instant response to the emotional state;
[0017] The adjustment module includes a predictive emotion adjustment module, an emotion resonance module and a deep learning module. The predictive emotion adjustment module collects the user's previous emotion state data to form a time series, applies a bidirectional LSTM network, inputs the past emotion state sequence, predicts future emotion trends, and calculates a smooth curve for content adjustment based on the predicted emotion trends to ensure that the emotion intensity change during the transition process does not exceed 0.05 per second. The emotion resonance module analyzes the group emotional response of users in similar situations, adjusts the content of individual users, and enhances the emotion resonance deep learning module. The DCCA network architecture is adopted. The deep learning module includes an input layer, a feature extraction layer, a fusion layer and a classification layer. The input layer multimodal data input includes facial expression images, speech waveforms and biological signals. The feature extraction layer adopts ResNet50 as a facial expression feature extractor, MFCC and Spectral Contrast are used for speech feature extraction, and a custom one-dimensional convolutional layer processes biological signals. The fusion layer fuses the features of different modalities through the attention mechanism to enhance the ability to capture key emotional clues. The classification layer uses a fully connected layer with a Softmax activation function to output the probability distribution of seven categories of emotions.
[0018] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:
[0019] The VR emotion recognition and response method based on multimodal fusion includes the following steps:
[0020] S1. Data Collection
[0021] Collect user facial expression images, use OpenCV library for facial detection, and combine FACS to mark key points of expression;
[0022] Collect user voice audio, read the audio using the SoundFile library, and extract audio features using the librosa library;
[0023] Collect user biometrics and use Arduino as a data collection platform to store signal data in real time;
[0024] S2. Data preprocessing
[0025] Use the conversion function in the Pillow library to convert the facial expression image to a grayscale image and scale the pixel values to the [0, 1] interval;
[0026] Use the VAD function in the pyAudioAnalysis library to achieve speech audio noise reduction and segmentation;
[0027] The Butterworth filter in the SciPy library is used for signal filtering, and the Resample library is used to ensure that all signals have the same sampling rate;
[0028] S3. Feature extraction
[0029] Load the pre-trained ResNet or VGG16 model in TensorFlow or PyTorch framework and extract feature vectors;
[0030] Use Wav2Vec or DeepSpeech models to extract intonation features;
[0031] Extract key physiological indicators from biological features through wavelet transform or fast Fourier transform;
[0032] S4. Real-time monitoring and dynamic adjustment
[0033] Classify the emotional state obtained after multimodal emotion fusion, and map the category to the response label according to the user's current emotional state;
[0034] Build a content library containing a variety of emotion-oriented content, preset a series of response rules, and adjust the content in real time according to the detected emotional state;
[0035] Continuously monitor the user's emotional state and adjust content based on real-time changes;
[0036] Collect user feedback on content adjustments, including direct evaluation and indirect behavioral data, and optimize content adaptation strategies based on user feedback;
[0037] S5. Emotion Recognition
[0038] After preprocessing and feature extraction, the real-time collected data stream is sent to the trained DCCA network;
[0039] Use sensors to capture user behavior data in real time and obtain original user data;
[0040] Clean the original user data, remove noise, and standardize the data format to ensure data quality;
[0041] Extract key features of processed raw user data through deep learning models;
[0042] The processed feature data is input into the trained deep conjugate component DCCA network in real time;
[0043] The emotional representation output by the network is converted into a probability distribution through the softmax layer to determine the most likely emotional category and its intensity. The softmax function is used to convert the emotional representation into the probability distribution of various emotions. The probability value of each emotional category reflects the possibility of its occurrence. The emotional category with the highest probability value is selected as the currently recognized emotional state, and the emotional intensity is judged based on the probability value.
[0044] S6. Response Generation
[0045] Categorize VR content by emotion type and intensity level, and design virtual scenes and interactive elements corresponding to each emotion;
[0046] S7. Response strategy formulation
[0047] Based on the results of emotion recognition, the brightness, color, sound effects and behavior of interactive objects in the VR scene are dynamically changed to enhance immersion and emotional resonance.
[0048] In one or more embodiments of the present invention, in step S4, it further includes:
[0049] The data collection and recording system needs to continuously collect user behavior data and combine it with the user emotional state data provided by the emotion recognition module to form a multi-dimensional user behavior profile;
[0050] Feature engineering and model training Based on the collected data, feature engineering is performed to extract key features that have a significant impact on user behavior prediction;
[0051] Create detailed user profiles that are dynamically updated over time to reflect changes in user preferences;
[0052] Design A / B tests to verify the effectiveness of personalization strategies. Based on the A / B test results, analyze the effectiveness and shortcomings of personalized recommendations, adjust model parameters, or optimize recommendation algorithms.
[0053] In one or more embodiments of the present invention, the user portrait includes the user's interest information, emotional tendency information, and behavior pattern information.
[0054] Compared with the prior art, the VR emotion recognition and response system and method based on multimodal fusion of the present invention realizes accurate recognition of the user's emotional state, significantly improves the emotional interactivity and personalization of the intangible cultural heritage VR experience, enhances the accuracy and robustness of emotion recognition through multimodal data fusion, improves the limitations of single modality recognition, and dynamically adjusts the ability of VR content, providing users with an immersive experience that is closer to personal emotional needs, promoting emotional resonance, and enhancing the dissemination effect of intangible cultural heritage.
[0055] By adopting the DCCA algorithm for multimodal data fusion, the technical difficulties of cross-modal emotion fusion and consistency calibration are solved, and more accurate emotion recognition is achieved, which brings a revolutionary improvement to the intangible cultural heritage VR experience. It has important academic value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0057] Figure 1 It is a flowchart of a VR emotion recognition and response system based on multimodal fusion in one embodiment of the present invention;
[0058] Figure 2 The flowchart of the VR emotion recognition and response method based on multimodal fusion in one embodiment of the present invention is as follows: Figure 1 ;
[0059] Figure 3 The flowchart of the VR emotion recognition and response method based on multimodal fusion in one embodiment of the present invention is as follows: Figure 2 . DETAILED DESCRIPTION
[0060] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0061] The VR emotion recognition and response system based on multimodal fusion in one embodiment of the present invention includes a data acquisition module, a data preprocessing module, a feature extraction module, a DCCA network module, an emotion recognition module and a response generation module. Among them, the data acquisition module is used to capture the user's body language feature signals, oral language feature signals and biometric feature signals in real time. Body language feature signals include movements of various parts of individuals such as facial expressions, eye movements, hand movements, and leg movements. Since body language can make different movements under different conditions, it can be used to replace language to achieve the purpose of expressing emotions. Therefore, the user's emotional state can be judged by analyzing the user's body language. Therefore, it is very necessary to collect the user's body language. Oral language feature signals include feature signals such as speech speed, intonation, and voice spoken orally, and the user's emotional state can be judged by combining oral language features. Biometric feature signals include biometric feature signals such as heartbeat, heart rate, and breathing. It is most intuitive to judge the user's emotional state through biometric feature signals. From the fundamental point of view, the accuracy of the VR emotion recognition and response system in ultimately judging the user's emotional state is improved.
[0062] Specifically, the data acquisition module includes a facial expression image acquisition module, a voice and audio acquisition module, and a biometric signal acquisition module. The facial expression image acquisition module is used to collect facial expressions, and uses the OpenCV library for facial detection, and combines FACS (Facial Action Coding System) to annotate the expression state. The voice and audio acquisition module is integrated on the VR device to collect the voice of the VR device user, and uses the SoundFile library to read the audio, and uses the librosa library to extract audio features. The biometric signal acquisition module is integrated on the VR device, using Arduino as a data acquisition platform to store signal data in real time.
[0063] The data preprocessing module preprocesses the data collected by the data acquisition module. It uses image graying, speech noise reduction, signal filtering and other technologies to process the data collected by the data acquisition module to ensure the quality of the data and reduce noise interference.
[0064] Specifically, the data preprocessing module includes an image processing module, an audio processing module, and a signal processing module. The image processing module uses the conversion function in the Pillow library to convert the image into a grayscale image and scale the pixel value to the interval [0, 1]. The same image format is convenient for subsequent processing. The audio processing module uses the VAD function in the pyAudioAnalysis library to achieve audio noise reduction and segmentation. The signal processing module uses the Butterworth filter in the SciPy library for signal filtering, and uses the Resample library to ensure that the signals have the same sampling rate.
[0065] The feature extraction module is used to extract the key features of the data processed by the data preprocessing module. In general, it uses pre-trained deep learning models and signal processing technology to efficiently extract key features from each modal data. The deep learning models are generally ResNet and VGG16. Specifically, the feature extraction module includes a facial feature extraction module, a voice tone feature extraction module, and a physiological feature extraction module. The facial feature extraction module uses the pre-trained ResNet or VGG16 model under the TensorFlow or PyTorch framework to extract feature vectors. The voice tone feature extraction module uses the Wav2Vec or DeepSpeech model to extract voice tone features. The voice tone features include pitch, rhythm, and intensity. The physiological feature extraction module extracts physiological indicators from biometric signals through wavelet transform or fast Fourier transform.
[0066] The DCCA network module builds a DCCA network to learn the deep associations between different modal data to ensure the consistency and integrity of emotional representation. The emotion recognition module analyzes multimodal data in real time based on the DCCA network module and outputs a unified emotional representation. The response generation module intelligently adjusts VR content based on the recognition results of the emotion recognition module to provide a personalized experience, such as showing a warm scene when the user feels sad and adding stimulating elements when the user is excited.
[0067] The VR emotion recognition and response system also includes an emotion-driven content adaptation module, which is used to adjust the content in the VR device according to the user's emotional state to enhance the user experience. Specifically, the content adaptation module includes a state classification module, a label mapping module, and a preset response module. The state classification module is used to classify the emotional state obtained after multimodal emotion fusion. The label mapping module maps the emotional state to the label according to the user's current emotional state. The preset response module establishes a content library containing a variety of emotion-oriented content, presets response rules based on the emotion-oriented content, and specifies how the system should adjust the content when a specific emotional state is detected.
[0068] The content adaptation module includes an adjustment module, which is used to monitor the user's emotional state and adjust the content according to the real-time changes in the user's emotional state. The adjustment module uses a deep learning model to continuously analyze the user's multimodal data, including facial expressions, voice and biological signals, to identify the emotional state in real time, set the threshold of emotional changes, monitor significant changes, and immediately trigger the content adjustment process. The adjustment module can also retrieve matching VR scenes, music and interactive elements based on the emotional tag system to achieve instant response to the emotional state.
[0069] Among them, the adjustment module includes a predictive emotion adjustment module, an emotion resonance module and a deep learning module. The predictive emotion adjustment module can collect the user's previous emotion state data to form a time series, apply a bidirectional LSTM network, input the past emotion state sequence, predict the future emotion trend, and calculate the smooth curve of content adjustment based on the predicted emotion trend to ensure that the change in emotion intensity during the transition process does not exceed 0.05 per second. The emotion resonance module is used to analyze the group emotional response of users in similar situations, adjust the content of individual users, and enhance emotional resonance. The deep learning module adopts the DCCA network architecture.
[0070] The deep learning module includes input layer, feature extraction layer, fusion layer and classification layer. The input layer uses multimodal data input, including facial expression images, speech waveforms and biological signals. The feature extraction layer uses ResNet50 as a facial expression feature extractor, MFCC and Spectral Contrast for speech feature extraction, and a custom one-dimensional convolutional layer to process biological signals. The fusion layer fuses features of different modalities through the attention mechanism to enhance the ability to capture key emotional clues. The classification layer uses a fully connected layer with a Softmax activation function to output the probability distribution of seven categories of emotions.
[0071] The VR emotion recognition and response system of the present invention realizes accurate recognition of the user's emotional state, significantly improving the emotional interactivity and personalization of the intangible cultural heritage VR experience. Through multimodal data fusion, the accuracy and robustness of emotion recognition are enhanced, and the limitations of single-modal recognition are improved. The ability to dynamically adjust VR content provides users with an immersive experience that is closer to their personal emotional needs, promotes emotional resonance, and enhances the dissemination effect of intangible cultural heritage.
[0072] The VR emotion recognition and response system of the present invention solves the technical problems of cross-modal emotion fusion and consistency calibration by adopting the DCCA algorithm for multimodal data fusion, realizes more accurate emotion recognition, brings revolutionary improvement to the intangible cultural heritage VR experience, and has important academic value and broad application prospects.
[0073] like Figure 1 As shown, a VR emotion recognition and response method based on multimodal fusion in one embodiment of the present invention includes the following steps:
[0074] S1. Data Collection
[0075] Collect user facial expression images, use the OpenCV library for facial detection, and combine FACS to annotate key points of expression, such as the degree of opening and closing of eyes and mouth. Generally, high frame rate (at least 30fps) infrared camera is used to collect user faces to adapt to image acquisition in low light environment.
[0076] To collect user voice audio, a dual-microphone array is used and integrated into VR devices to ensure sound positioning accuracy. The SoundFile library is used to read audio and the librosa library is used to extract audio features.
[0077] Collect user biometrics, embed heart rate, skin conductance and respiration monitoring sensors into VR devices, and use Arduino as a data acquisition platform to store signal data in real time.
[0078] S2. Data preprocessing
[0079] Use the conversion function in the Pillow library to convert the facial expression image into a grayscale image and scale the pixel values to the [0, 1] interval to facilitate subsequent neural network processing.
[0080] Use the VAD (VoiceActivity Detection) function in the pyAudioAnalysis library to achieve voice audio noise reduction and segmentation;
[0081] The Butterworth filter in the SciPy library is used for signal filtering, and the Resample library is used to ensure that all signals have the same sampling rate;
[0082] S3. Feature extraction
[0083] Load the pre-trained ResNet or VGG16 model in TensorFlow or PyTorch framework, extract feature vectors, and implement facial feature extraction and pre-training models;
[0084] Use Wav2Vec or DeepSpeech model to extract intonation features, including pitch, rhythm, and intensity, and generate a speech model;
[0085] Through wavelet transform or fast Fourier transform (FFT), key physiological indicators are extracted from biological features for signal analysis;
[0086] S4. Real-time monitoring and dynamic adjustment
[0087] like Figure 2 As shown in the figure, the emotional state obtained after multimodal emotion fusion is classified into basic emotional categories such as happiness, sadness, surprise, anger, fear, disgust, neutrality, etc. Then, according to the user's current emotional state, the category is mapped to the corresponding label so that the subsequent VR content adjustment can refer to these labels.
[0088] Build a content library containing a variety of emotion-oriented content, such as music, videos, story lines, and environmental settings for different emotional states. At the same time, preset a series of response rules to specify how the system should adjust the content when a specific emotional state is detected. For example, when a user shows sadness, the system may automatically play soft music, display warm pictures, or start an encouraging conversation to help the user regulate his or her emotions.
[0089] Real-time monitoring and dynamic adjustment During the VR experience, the system continuously monitors the user's emotional state and quickly adjusts the content based on real-time changes. This requires the system to have a high degree of real-time processing capabilities and a flexible response mechanism. For example, if it is detected that the user changes from sadness to joy, the system should quickly switch to more lively and cheerful content to match the user's emotional changes.
[0090] Over time, the system should be able to learn and understand each user's specific emotional preferences and reaction patterns through machine learning algorithms. This personalized learning helps the system more accurately predict the user's possible emotional changes and prepare corresponding content adjustments in advance, thereby providing a more personalized experience.
[0091] The data collection and recording system needs to continuously collect user behavior data, including but not limited to the user's interaction frequency, selection preferences, stay time, reaction speed, etc. in the virtual environment. At the same time, combined with the user's emotional state data provided by the emotion recognition module, a multi-dimensional user behavior profile is formed. These data should be anonymized to protect user privacy.
[0092] Feature engineering and model training Based on the collected data, feature engineering is performed to extract key features that have a significant impact on user behavior prediction. Personalized recommendation models are trained using machine learning or deep learning algorithms, such as neural networks, decision trees, support vector machines, etc. The goal of the model is to predict the user's content preferences under different emotional states, as well as the impact of content adjustments on user emotions.
[0093] Create detailed user portraits that include information about the user's interests, hobbies, emotional tendencies, behavioral patterns, etc. User portraits should be dynamically updated over time to reflect changes in user preferences. This helps the system understand each user more accurately and provides a basis for personalized content recommendations.
[0094] Experimental design and A / B testing In order to verify the effectiveness of the personalization strategy, the A / B testing method can be used. Users are randomly divided into two groups, one as the experimental group, receiving personalized content recommendations; the other as the control group, receiving general or random content. Compare the emotional changes, participation, satisfaction and other indicators of the two groups of users during the experience process to evaluate the actual effect of the personalization strategy. Strategy iteration and optimization According to the A / B test results, analyze the effectiveness and shortcomings of personalized recommendations, adjust model parameters or optimize the recommendation algorithm. This is an ongoing process. With the accumulation of data and the maturity of the model, the accuracy and efficiency of personalized recommendations will gradually improve.
[0095] Anomaly Detection and Intervention In the process of personalized learning, the system should have the ability to detect anomalies and promptly identify and handle possible deviations or abnormal situations, such as sudden changes in user behavior, large model prediction errors, etc. If necessary, human intervention is carried out to ensure that the system runs stably and the user experience is not negatively affected.
[0096] Collect user feedback on content adjustments, including direct evaluations and indirect behavioral data, such as the length of time spent on specific content, the number of repeat visits, etc. Through a continuous feedback loop, the system can continuously optimize content adaptation strategies, improve the accuracy of emotional responses, and improve user experience satisfaction.
[0097] In the process of implementing emotion-driven content adaptation, privacy protection principles must be strictly followed to ensure that all collected data is agreed to by the user and is only used for the purpose of improving the experience. In addition, ethical issues must be considered to avoid adverse effects of content adjustment on users, such as overstimulation or discomfort.
[0098] S5. Emotion Recognition
[0099] like Figure 3 As shown, the real-time collected data stream is sent to the trained DCCA network after preprocessing and feature extraction;
[0100] Specifically, sensors are used to capture user behavior data in real time to obtain original user data; sensors such as cameras, microphones, and biosignal sensors are used to capture user behavior data in real time, including facial expressions, voice intonation, heart rate, skin conductance and other physiological signals.
[0101] Clean the original user data, remove noise, and standardize the data format to ensure data quality;
[0102] Deep learning models such as convolutional neural networks (CNN) for facial expression recognition, recurrent neural networks (RNN) for speech intonation analysis, and specialized biosignal processing algorithms are used to extract key features. The processed feature data is fed into the trained deep conjugate component analysis (DCCA) network in real time. The sentiment representation output by the network is converted into a probability distribution through a softmax layer to determine the most likely sentiment category and its intensity. The DCCA network outputs a high-dimensional vector that contains a comprehensive representation of the sentiment state. The sentiment representation is converted into a probability distribution of various sentiments using the softmax function, and the probability value of each sentiment category reflects the likelihood of its occurrence. The sentiment category with the highest probability value is selected as the currently recognized sentiment state, and the sentiment intensity is judged based on the probability value.
[0103] S6. Response Generation
[0104] Categorize VR content by emotion type and intensity level, such as calm, excitement, fear, etc. Design virtual scenes and interactive elements corresponding to each emotion.
[0105] S7. Response strategy formulation
[0106] Develop response strategies based on rules or machine learning, such as emotion adaptation algorithms based on decision trees. Real-time content adjustment: Dynamically change the brightness, color, sound effects, and behavior of interactive objects in VR scenes based on the results of emotion recognition to enhance immersion and emotional resonance.
[0107] The VR emotion recognition and response method of the present invention realizes accurate recognition of the user's emotional state, significantly improving the emotional interactivity and personalization of the intangible cultural heritage VR experience. Through multimodal data fusion, the accuracy and robustness of emotion recognition are enhanced, and the limitations of single modality recognition are improved. The ability to dynamically adjust VR content provides users with an immersive experience that is closer to their personal emotional needs, promotes emotional resonance, and enhances the dissemination effect of intangible cultural heritage.
[0108] The VR emotion recognition and response method of the present invention solves the technical problems of cross-modal emotion fusion and consistency calibration by adopting the DCCA algorithm for multimodal data fusion, realizes more accurate emotion recognition, brings revolutionary improvement to the intangible cultural heritage VR experience, and has important academic value and broad application prospects.
[0109] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
[0110] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
Claims
1. VR emotion recognition and response system based on multimodal fusion, characterized by: include A data acquisition module, used to capture the user's body language characteristic signals, oral language characteristic signals and biometric characteristic signals in real time; A data preprocessing module, used for preprocessing the data collected by the data collection module; A feature extraction module is used to extract key features of the data processed by the data preprocessing module; The DCCA network module is used to learn the deep associations between different modal data to ensure the consistency and integrity of sentiment representation; Emotion recognition module, which analyzes multimodal data in real time based on the DCCA network module and outputs a unified emotion representation; The response generation module intelligently adjusts the VR content based on the recognition results of the emotion recognition module to provide a personalized experience.
2. The VR emotion recognition and response system based on multimodal fusion according to claim 1 is characterized in that: The data acquisition module comprises: The facial expression image acquisition module is used to collect facial expressions, use the OpenCV library for facial detection, and combine FACS to annotate the expression status; The voice audio acquisition module is integrated on the VR device to collect the voice of the VR device user, and uses the SoundFile library to read the audio and the librosa library to extract audio features; The biometric signal acquisition module is integrated on the VR device, uses Arduino as a data acquisition platform, and stores signal data in real time.
3. The VR emotion recognition and response system based on multimodal fusion according to claim 1 is characterized in that: The data preprocessing module comprises: The image processing module uses the conversion function in the Pillow library to convert the image into a grayscale image and scale the pixel values to the range [0, 1]; Audio processing module, using the VAD function in the pyAudioAnalysis library to achieve audio noise reduction and segmentation; In the signal processing module, the Butterworth filter in the SciPy library is used to filter the signal, and the Resample library is used to ensure that the signals have the same sampling rate.
4. The VR emotion recognition and response system based on multimodal fusion according to claim 1, characterized in that: The feature extraction module comprises: Facial feature extraction module, using ResNet or VGG16 models pre-trained in TensorFlow or PyTorch framework to extract feature vectors; Intonation feature extraction module, which uses Wav2Vec or DeepSpeech model to extract intonation features, including pitch, rhythm and intensity; The physiological feature extraction module extracts physiological indicators from biometric signals through wavelet transform or fast Fourier transform.
5. The VR emotion recognition and response system based on multimodal fusion according to claim 1, characterized in that: It includes an emotion-driven content adaptation module, which is used to adjust the content in the VR device according to the user's emotional state and improve the user experience.
6. The VR emotion recognition and response system based on multimodal fusion according to claim 5 is characterized in that: The content adaptation module includes: The state classification module is used to classify the emotional state obtained after multimodal emotion fusion; The label mapping module maps the emotional state to the label according to the user's current emotional state; The preset response module establishes a content library containing a variety of emotion-oriented content, presets response rules based on the emotion-oriented content, and stipulates how the system should adjust the content when a specific emotional state is detected.
7. The VR emotion recognition and response system based on multimodal fusion according to claim 5 or 6, characterized in that: The content adaptation module includes: An adjustment module is used to monitor the user's emotional state and adjust the content according to the real-time changes in the user's emotional state; The adjustment module uses deep learning models to continuously analyze multimodal user data, including facial expressions, voice, and biological signals, to identify emotional states in real time, set emotional change thresholds, and immediately trigger content adjustment processes when significant changes are detected. Based on the emotional tags, the system retrieves matching VR scenes, music, and interactive elements to achieve instant response to emotional states; The adjustment module comprises: The predictive emotion adjustment module collects the user's previous emotion state data to form a time series, applies a bidirectional LSTM network, inputs the past emotion state sequence, predicts the future emotion trend, and calculates the smooth curve of content adjustment based on the predicted emotion trend to ensure that the change in emotion intensity during the transition process does not exceed 0.05 per second; The emotional resonance module analyzes the group emotional reactions of users in similar situations, adjusts the content of individual users, and enhances emotional resonance; The deep learning module adopts the DCCA network architecture. The deep learning module includes: Input layer, multimodal data input, including facial expression images, speech waveforms, and biological signals; Feature extraction layer, using ResNet50 as facial expression feature extractor, MFCC and Spectral Contrast for speech feature extraction, and a custom one-dimensional convolutional layer to process biological signals; The fusion layer combines the features of different modalities through the attention mechanism to enhance the ability to capture key emotional clues; The classification layer uses a fully connected layer with a Softmax activation function to output the probability distribution of seven categories of emotions.
8. VR emotion recognition and response method based on multimodal fusion, characterized in that: The following steps are involved: S1. Data Collection Collect user facial expression images, use OpenCV library for facial detection, and combine FACS to mark key points of expression; Collect user voice audio, read the audio using the SoundFile library, and extract audio features using the librosa library; Collect user biometrics and use Arduino as a data collection platform to store signal data in real time; S2. Data preprocessing Use the conversion function in the Pillow library to convert the facial expression image to a grayscale image and scale the pixel values to the [0, 1] interval; Use the VAD function in the pyAudioAnalysis library to achieve speech audio noise reduction and segmentation; The Butterworth filter in the SciPy library is used for signal filtering, and the Resample library is used to ensure that all signals have the same sampling rate; S3. Feature extraction Load the pre-trained ResNet or VGG16 model in TensorFlow or PyTorch framework and extract feature vectors; Use Wav2Vec or DeepSpeech models to extract intonation features; Extract key physiological indicators from biological features through wavelet transform or fast Fourier transform; S4. Real-time monitoring and dynamic adjustment Classify the emotional state obtained after multimodal emotion fusion, and map the category to the response label according to the user's current emotional state; Build a content library containing a variety of emotion-oriented content, preset a series of response rules, and adjust the content in real time according to the detected emotional state; Continuously monitor the user's emotional state and adjust content based on real-time changes; Collect user feedback on content adjustments, including direct evaluation and indirect behavioral data, and optimize content adaptation strategies based on user feedback; S5. Emotion Recognition After preprocessing and feature extraction, the real-time collected data stream is sent to the trained DCCA network; Use sensors to capture user behavior data in real time and obtain original user data; Clean the original user data, remove noise, and standardize the data format to ensure data quality; Extract key features of processed raw user data through deep learning models; The processed feature data is input into the trained deep conjugate component DCCA network in real time; The emotional representation output by the network is converted into a probability distribution through the softmax layer to determine the most likely emotional category and its intensity. The softmax function is used to convert the emotional representation into the probability distribution of various emotions. The probability value of each emotional category reflects the possibility of its occurrence. The emotional category with the highest probability value is selected as the currently recognized emotional state, and the emotional intensity is judged based on the probability value. S6. Response Generation Categorize VR content by emotion type and intensity level, and design virtual scenes and interactive elements corresponding to each emotion; S7. Response strategy formulation Based on the results of emotion recognition, the brightness, color, sound effects and behavior of interactive objects in the VR scene are dynamically changed to enhance immersion and emotional resonance.
9. The VR emotion recognition and response method based on multimodal fusion according to claim 8 is characterized in that: In the step S4, it also includes: The data collection and recording system needs to continuously collect user behavior data and combine it with the user emotional state data provided by the emotion recognition module to form a multi-dimensional user behavior profile; Feature engineering and model training Based on the collected data, feature engineering is performed to extract key features that have a significant impact on user behavior prediction; Create detailed user profiles that are dynamically updated over time to reflect changes in user preferences; Design A / B tests to verify the effectiveness of personalization strategies. Based on the A / B test results, analyze the effectiveness and shortcomings of personalized recommendations, adjust model parameters, or optimize recommendation algorithms.
10. The VR emotion recognition and response method based on multimodal fusion according to claim 9, characterized in that: The user portrait includes the user's interest information, emotional tendency information, and behavior pattern information.
Citation Information
Cited By
Offline network customer service quality inspection method and device, electronic equipment and storage medium
CN119721870A
Emotion exchange method based on emotion recognition
CN119770044A
Animal language conversion method and device, electronic equipment and storage medium
CN119943059A
Mental health early warning method and system for mental health education
CN120148769A
Large language model driven VR intelligent role emotion feedback method and device
CN121277361A
