Multi-mode XR emotion interaction method, system and device and storage medium

Abstract

The invention discloses a multi-modal XR emotion interaction method, system and device and a storage medium. The method comprises the steps of collecting multi-modal data of a user through a multi-modal perception interface of an XR device; preprocessing the collected multi-modal data, and performing emotion recognition on the preprocessed multi-modal data by adopting a deep neural network to obtain an emotion recognition result; constructing an emotion knowledge base, mapping the emotion recognition result into the emotion knowledge base, and reasoning and analyzing the current emotion; and automatically classifying the emotion according to the personalized emotion model of the user, and intervening and regulating the emotion of the user according to the current emotion of the user, the personalized emotion model and an emotion regulation reinforcement learning algorithm. Omnibearing, accurate, dynamic and personalized perception and quantitative analysis of the emotional state of the user are realized, and the emotional interaction experience in an XR scene can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal XR emotion interaction method, system, device and storage medium. Background Art

[0002] With the rapid development of communication technology and virtual reality (XR) technology, remote space virtual communication will gradually become an important way for people to work, study and socialize. Compared with traditional face-to-face communication, virtual communication breaks the limitations of time and space and greatly improves the convenience and flexibility of communication. However, the current XR system still has some shortcomings in emotional interaction. The user's emotional experience in the XR environment is often not real, rich and personalized enough, and the system lacks the ability to intelligently perceive and actively adjust the user's emotional state, which limits the application potential of XR in the field of emotional computing.

[0003] Specifically, some existing XR systems use computer vision algorithms to recognize users' facial expressions and map them to the facial animations of virtual avatars, thereby realizing real-time expression of avatar emotions. However, this method can usually only recognize a limited number of basic expressions, and it is difficult to capture the subtle changes and complexity of users' emotions. In addition, facial expressions do not always fully reflect a person's true emotional state. In addition, in the prior art, user emotion recognition technology based on speech and semantic analysis can also judge the user's emotional state to a certain extent by analyzing the user's voice features (such as pitch, volume, etc.) and semantic content. However, it is difficult to fully and accurately reflect the user's overall emotional state by relying solely on speech and semantic information, especially when the user does not speak or speaks very little. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a multimodal XR emotion interaction method, system, device and storage medium to achieve accurate perception, natural expression, active regulation and intelligent feedback of the user's emotional state.

[0005] The first embodiment of the present invention provides a multimodal XR emotion interaction method, which includes: collecting multimodal data of a user through a multimodal perception interface of an XR device; preprocessing the collected multimodal data, using a deep neural network to perform emotion recognition on the preprocessed multimodal data, and obtaining emotion recognition results; constructing an emotion knowledge base, mapping the emotion recognition results to the emotion knowledge base to reason and analyze current emotions; automatically classifying emotions according to the user's personalized emotion model, and intervening and regulating the user's emotions according to the user's current emotions, the personalized emotion model, and the emotion regulation reinforcement learning algorithm.

[0006] Optionally, the multimodal data includes the user's facial expressions, voice data, head posture, gesture movements, the user's gaze point and line of sight movement trajectory, and the user's physiological signals.

[0007] Optionally, the step of preprocessing the collected multimodal data includes: converting the multimodal data into a unified digital form; performing noise reduction, smoothing and normalization on the converted multimodal data; aligning the timestamps of the processed multimodal data and extracting key features that can reflect the emotional state; and fusing the key features according to a predefined data structure to form a unified multimodal emotion feature vector.

[0008] Optionally, the multimodal data are normalized using a Z-score standardization method.

[0009] Optionally, the step of using a deep neural network to perform emotion recognition on the preprocessed multimodal data and obtaining the emotion recognition result includes: inputting the modal emotion feature vector into the deep neural network to obtain the probability distribution of each type of emotion; according to the probability distribution of each type of emotion, combined with the attention mechanism, dynamically adjusting the weights of different modalities to give the final emotion recognition result.

[0010] The second embodiment of the present invention provides a multimodal XR emotion interaction system, which includes: a data acquisition module, which collects multimodal data of a user through a multimodal perception interface of an XR device; an emotion recognition module, which preprocesses the collected multimodal data, uses a deep neural network to perform emotion recognition on the preprocessed multimodal data, and obtains emotion recognition results; an emotion understanding module, which constructs an emotion knowledge base, maps the emotion recognition results to the emotion knowledge base, and performs reasoning and analysis on current emotions; an emotion regulation module, which automatically classifies emotions according to a personalized emotion model of the user, and intervenes and regulates the user's emotions according to the user's current emotions, the personalized emotion model, and the emotion regulation reinforcement learning algorithm.

[0011] Optionally, the data acquisition module includes: a camera for capturing the user's facial expressions;

[0012] Microphone, used to collect the user's voice signal; tracking sensors, including head tracking sensors and hand tracking sensors, used to collect the user's head posture and gesture movements; eye tracking module, used to collect the user's gaze point and line of sight movement trajectory; wearable sensors, used to collect the user's physiological signals.

[0013] Optionally, the emotion recognition module is also used to: convert multimodal data into a unified digital form; perform noise reduction, smoothing and normalization on the converted multimodal data; timestamp align the processed multimodal data and extract key features that can reflect the emotional state; and fuse the key features according to a predefined data structure to form a unified multimodal emotion feature vector.

[0014] A third embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of any of the multimodal XR emotion interaction methods described above are implemented.

[0015] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of any of the multimodal XR emotion interaction methods described above.

[0016] In the technical solution provided by the embodiment of the present invention, the multi-sensory interface of the XR device is utilized to collect and integrate multimodal data including the user's facial expressions, voice, movements, and line of sight in real time. Deep learning, knowledge graphs, reinforcement learning, etc. are combined to build a personalized user emotion model to achieve accurate perception, natural expression, active regulation, and intelligent feedback of the user's emotional state, which can significantly enhance the emotional interaction experience in the XR scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of one embodiment of the multimodal XR emotion interaction method of the present invention.

[0018] Figure 2 A schematic diagram of a flow chart of another embodiment of the multimodal XR emotion interaction method of the present invention.

[0019] Figure 3 Schematic diagram of the emotion recognition process of the multimodal XR emotion interaction method of the present invention.

[0020] Figure 4 A schematic diagram of the flow of emotion intervention and regulation of the multimodal XR emotion interaction method of the present invention.

[0021] Figure 5 It is a schematic diagram of the structure of the multimodal XR emotion interaction system of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] Please refer to Figure 1 As shown, the present invention provides a multimodal XR emotion interaction method, the method comprising the following steps:

[0025] Step S10: Collect multimodal data of the user through the multimodal perception interface of the XR device.

[0026] The multimodal data includes the user's facial expressions, voice data, head posture, hand gestures, the user's gaze point and line of sight movement trajectory, and the user's physiological signals.

[0027] Specifically, the present invention uses the camera of the XR device to capture the changes in the user's facial expression, and uses the microphone to collect the user's voice signal; uses the head and hand tracking sensors of the XR device to collect the user's head posture and gesture movements; uses the eye tracking module to collect the user's gaze point and line of sight movement trajectory; and uses the wearable sensor connected to the XR device to collect the user's heart rate, skin electricity and other physiological signals. The present invention aligns the data collected by different sensors to ensure the time consistency of the data.

[0028] The present invention comprehensively collects the user's emotional data in the XR environment, makes up for the limitations of single-modality emotion recognition, and provides a more accurate and reliable basis for emotion analysis.

[0029] Step S20, preprocessing the collected multimodal data, using a deep neural network to perform emotion recognition on the preprocessed multimodal data, and obtaining an emotion recognition result.

[0030] In one embodiment of the present invention, please refer to Figure 2 , Figure 3 As shown, the step of preprocessing the collected multimodal data includes:

[0031] Step S21, converting the multimodal data into a unified digital form.

[0032] Specifically, data of different modalities (such as image data, audio data, text data, etc.) are converted into a unified digital representation.

[0033] Step S22, performing noise reduction, smoothing and normalization processing on the converted multimodal data.

[0034] The Z-score normalization method is used to normalize the multimodal data to eliminate the differences in scale and distribution of different features.

[0035] Step S23, aligning the timestamps of the processed multimodal data and extracting key features that can reflect the emotional state.

[0036] The present invention performs timestamp alignment on data collected by different sensors to ensure synchronization of multimodal data in the time dimension. And for data of different modes, key features that can reflect emotional states, such as facial key points, acoustic features, movement speed, etc. are extracted. Specifically, feature extraction models are designed for facial key points, acoustic features, movement features, etc. For example, CNN is used to extract features of facial expressions, and LSTM is used to extract features of speech. Then the key features extracted by each modality are fused and sent to a jointly trained deep neural network, and the output of the network is the probability distribution of various emotions. By selecting the category with the highest probability, a coarse-grained classification result of the user's current emotion (such as happiness, sadness, anger, etc.) can be obtained.

[0037] Step S24, the key features are fused according to a predefined data structure to form a unified multimodal emotion feature vector. Specifically, the standardized multimodal features are fused according to a predefined data structure to form a unified multimodal emotion feature vector.

[0038] The present invention achieves refined and multi-granular emotion understanding through deep learning, which can not only determine the category of emotions, but also analyze their intensity. Multimodal fusion further improves the robustness of emotion prediction.

[0039] One embodiment of the present invention further includes an emotion data storage module, which is used to design a standardized storage format and database model for emotion data, such as defining fields such as emotion category, intensity, timestamp, and associated events. The multimodal emotion feature vector is stored in the database together with the corresponding metadata (such as user ID, scene information, etc.) to construct a structured emotion archive. The database is indexed to support fast retrieval and query.

[0040] The data collation and standardized storage of the present invention improves the accessibility, usability and interoperability of emotion data, provides high-quality data support for emotion analysis and personalized services, and also lays a data foundation for long-term tracking of user emotions and mental health analysis.

[0041] In one embodiment of the present invention, the step of using a deep neural network to perform emotion recognition on the preprocessed multimodal data and obtaining the emotion recognition result comprises:

[0042] Input the multimodal emotion feature vector into the deep neural network to obtain the probability distribution of various emotions;

[0043] According to the probability distribution of various emotions and combined with the attention mechanism, the weights of different modalities are dynamically adjusted to give the final emotion recognition result.

[0044] Step S30, construct an emotion knowledge base, map the emotion recognition results to the emotion knowledge base to reason and analyze the current emotion. Specifically, collect emotion theories and models in the fields of psychology and cognitive science, and construct a structured emotion knowledge base, which includes emotion concepts, the relationship between emotions, the relationship between emotions and behaviors and situations, etc.

[0045] The emotion labels output by the emotion recognition module are mapped to the emotion knowledge base, and through knowledge reasoning, the causes of the user's current emotions, possible changing trends, and the impact on other emotions and behaviors are analyzed.

[0046] The present invention utilizes external knowledge to enhance the emotion analysis capability, so that the system has the ability to analyze emotions from multiple angles such as causes, developments, and impacts, making emotion perception more comprehensive and thorough.

[0047] Step S40, automatically classify emotions according to the user's personalized emotion model, and intervene and regulate the user's emotions according to the user's current emotions, personalized emotion model and emotion regulation reinforcement learning algorithm. For details, please refer to Figure 4 shown.

[0048] Personalized emotion model: Continuously track users' emotional response patterns in different situations, and use transfer learning and other technologies to generate personalized emotion models adapted to specific users based on the general emotion model.

[0049] The present invention dynamically adjusts the threshold and decision boundary of the emotion classifier according to the user's personalized emotion model to improve the accuracy of emotion classification. A personalized model is established for each user, which significantly improves the accuracy of emotion analysis and prediction. The active regulation strategy can be different for each user, and the effect is more significant.

[0050] The present invention independently decides which intervention measures to take (such as recommending music that relieves stress) based on the user's current emotional state and personalized emotional model, combined with the emotional regulation reinforcement learning algorithm, to guide the user's emotions in a positive direction. Among them, the emotional regulation reinforcement learning algorithm is to model the problem as a Markov decision process (MDP):

[0051] State space S: represents the user's emotional state, which can be represented by the results of emotion classification (such as happiness, sadness, etc.);

[0052] Action space A: represents the emotion regulation measures that the system can take, such as recommending music, adjusting scene tones, etc.

[0053] Reward function R: Gives numerical reward feedback based on the degree of improvement in the user's emotional state after taking measures;

[0054] State transfer function P: predicts the probability distribution of the user's emotional state at the next moment based on the current state and the action taken.

[0055] The present invention also collects user feedback on system control measures and continuously optimizes the control strategy through incremental learning.

[0056] In one embodiment of the present invention, an emotion-aware virtual assistant is also included, including emotional dialogue generation, mental health assessment, and XR scene construction, as follows:

[0057] Emotional dialogue generation: Based on a pre-trained large-scale language model, it is fine-tuned on emotional chat corpus to generate empathetic and emotional dialogue responses.

[0058] Mental health assessment: Design a mental health assessment scale based on questionnaires and behavioral analysis to periodically assess the user's mental health level and form a mental health report.

[0059] XR scene construction: Automatically build immersive XR interactive scenes for different emotion regulation needs, and optimize scene parameters in real time based on the user's physiological and behavioral feedback.

[0060] The present invention sets up an emotion-aware virtual assistant and adopts an anthropomorphic and emotional interaction method to make users more willing to open their hearts, which helps to relieve anxiety and obtain psychological support.

[0061] The present invention realizes the all-round, accurate, dynamic and personalized perception and quantitative analysis of the user's emotional state, which can significantly improve the emotional interaction experience in the XR scene. By constructing a closed-loop emotional perception-analysis-regulation-feedback framework, an end-to-end emotional intelligent interaction system is formed, providing a complete emotional computing middleware for XR application development. It innovatively combines mental health services with XR technology, and uses immersive experience and intelligent assistants to provide users with more intuitive and effective emotional management and psychological counseling. It creates a harmonious and natural emotional interaction atmosphere for multi-user XR systems, which helps to enhance empathy and social connections between users.

[0062] The present invention also provides a multimodal XR emotion interaction system, please refer to Figure 5 As shown, the system includes a data acquisition module 100, an emotion recognition module 200, an emotion understanding module 300 and an emotion regulation module 400. The data acquisition module 100 collects the user's multimodal data through the multimodal perception interface of the XR device. The emotion recognition module 200 preprocesses the collected multimodal data, uses a deep neural network to perform emotion recognition on the preprocessed multimodal data, and obtains emotion recognition results; the emotion understanding module 300 builds an emotion knowledge base, maps the emotion recognition results to the emotion knowledge base, and infers and analyzes the current emotions; the emotion regulation module 400 automatically classifies emotions according to the user's personalized emotion model, and intervenes and regulates the user's emotions according to the user's current emotions, personalized emotion model and emotion regulation reinforcement learning algorithm.

[0063] Specifically, the present invention uses the camera of the XR device to capture the changes in the user's facial expression, and uses the microphone to collect the user's voice signal; uses the head and hand tracking sensors of the XR device to collect the user's head posture and gesture movements; uses the eye tracking module to collect the user's gaze point and line of sight movement trajectory; and uses the wearable sensor connected to the XR device to collect the user's heart rate, skin electricity and other physiological signals. The present invention aligns the data collected by different sensors to ensure the time consistency of the data.

[0064] The present invention comprehensively collects the user's emotional data in the XR environment, makes up for the limitations of single-modality emotion recognition, and provides a more accurate and reliable basis for emotion analysis.

[0065] In one embodiment of the present invention, the emotion recognition module 200 is further used for:

[0066] Convert multimodal data into a unified digital form. Specifically, convert data of different modalities (such as image data, audio data, text data, etc.) into a unified digital representation.

[0067] Perform noise reduction, smoothing and normalization processing on the converted multimodal data;

[0068] Align the timestamps of the processed multimodal data and extract key features that can reflect the emotional state;

[0069] The key features are fused according to a predefined data structure to form a unified multimodal emotion feature vector.

[0070] The present invention adopts the Z-score standardization method to normalize multimodal data. To eliminate the differences in scale and distribution of different features. The data collected by different sensors are time-stamped to ensure the synchronization of multimodal data in the time dimension. And for data of different modes, key features that can reflect emotional states, such as facial key points, acoustic features, movement speed, etc. are extracted. Specifically, feature extraction models are designed for facial key points, acoustic features, movement features, etc. For example, CNN is used to extract the features of facial expressions, and LSTM is used to extract the features of speech. Then the key features extracted by each modality are fused and sent to the deep neural network of joint training. The output of the network is the probability distribution of various emotions. By selecting the category with the highest probability, the coarse-grained classification result of the user's current emotion (such as happiness, sadness, anger, etc.) can be obtained.

[0071] The present invention designs corresponding deep neural networks for different modal data, extracts high-level emotional features, realizes multi-granular analysis of emotion category recognition and intensity estimation, and finely characterizes the user's emotional state. The attention mechanism is used for multi-modal feature fusion to improve the accuracy and robustness of emotion recognition.

[0072] The present invention connects the emotion features extracted by deep learning with the knowledge graph by constructing an emotion knowledge graph (emotion knowledge base), and realizes knowledge-based emotion causal reasoning, trend prediction and impact analysis. It uses external knowledge to enhance the interpretability of emotion analysis and make it more in line with human intuition.

[0073] The present invention continuously tracks the user's emotional patterns, builds a personalized emotional model, and adaptively adjusts the emotion recognition strategy. Combined with reinforcement learning, it autonomously explores and optimizes the emotion regulation strategy to achieve targeted emotional intervention. Personalized modeling and intelligent regulation provide an emotional interaction solution that teaches students in accordance with their aptitude and improves the user experience.

[0074] The present invention combines a personalized emotion model with a virtual assistant to set up an emotional intelligent assistant. Through technologies such as speech synthesis and expression animation, the virtual assistant has anthropomorphic emotional interaction capabilities. The assistant can perceive the user's emotional state in real time, provide targeted emotional counseling dialogues, and create a friendly and natural interactive atmosphere. At the same time, the virtual assistant can guide the user to perform decompression activities in the XR environment, such as meditation and relaxation training. Biofeedback and neurofeedback technologies are integrated in the XR scene to achieve real-time interaction between the user's physiological state and the virtual environment. When the user is anxious or stressed, the system can automatically adjust the virtual scene (such as reducing saturation and slowing down the rhythm) to create an audio-visual environment that helps to relax. Physiological and EEG feedback data can also be used to evaluate the effect of emotion regulation training. Through the virtual assistant, users' subjective reports on their emotional state and stress level are regularly collected, and combined with objective physiological and behavioral data, the user's mental health status is evaluated. The system can use functions such as questionnaires and emotional diaries to understand the user's thoughts at any time. Once a significant negative change in the user's emotional state is detected, the system can automatically warn and recommend professional mental health service resources to the user.

[0075] In one embodiment of the present invention, a human-computer interaction optimization module is also included, which is mainly responsible for optimizing the human-computer interaction experience of the system. By dynamically adjusting the interaction strategy and presentation mode, real-time adaptation of emotional perception and interaction is achieved. At the same time, an intuitive and friendly user interface and interaction mode are designed to reduce the difficulty of learning and using the user and improve the usability and ease of use of the system. Through human-computer interaction optimization, a more natural and smooth emotional interaction experience is provided to the user.

[0076] The above is a schematic scheme of a multimodal XR emotion interaction system of the present invention. It should be noted that the technical scheme of the multimodal XR emotion interaction system and the technical scheme of the multimodal XR emotion interaction method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal XR emotion interaction system can be referred to the description of the technical scheme of the multimodal XR emotion interaction method mentioned above.

[0077] The present invention also provides a computing device, including but not limited to a memory and a processor, the processor and the memory are connected via a bus, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the multimodal instruction interaction method described in any of the above embodiments are implemented.

[0078] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned multimodal instruction interaction method.

[0079] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimodal instruction interaction method described above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the multimodal interaction method described above.

[0080] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned multimodal instruction interaction method.

[0081] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the multimodal instruction interaction method described above are of the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the multimodal instruction interaction method described above.

[0082] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0084] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal XR emotion interaction method, characterized in that: The method comprises: Collect the user's multimodal data through the multimodal perception interface of the XR device; Preprocess the collected multimodal data, use a deep neural network to perform emotion recognition on the preprocessed multimodal data, and obtain emotion recognition results; Constructing an emotion knowledge base, mapping the emotion recognition results into the emotion knowledge base to reason and analyze the current emotion; Emotions are automatically classified according to the user's personalized emotion model, and the user's emotions are intervened and regulated based on the user's current emotions, personalized emotion model and emotion regulation reinforcement learning algorithm.

2. The multimodal XR emotion interaction method according to claim 1, characterized in that: The multimodal data includes the user's facial expressions, voice data, head posture, hand gestures, the user's gaze point and line of sight movement trajectory, and the user's physiological signals.

3. The multimodal XR emotion interaction method according to claim 1, characterized in that: The step of preprocessing the collected multimodal data includes: Convert multimodal data into a unified digital form; Perform noise reduction, smoothing and normalization processing on the converted multimodal data; Align the timestamps of the processed multimodal data and extract key features that can reflect the emotional state; The key features are fused according to a predefined data structure to form a unified multimodal emotion feature vector.

4. The multimodal XR emotion interaction method according to claim 3, characterized in that: The Z-score standardization method was used to normalize the multimodal data.

5. The multimodal XR emotion interaction method according to claim 3, characterized in that: The step of using a deep neural network to perform emotion recognition on the preprocessed multimodal data and obtaining the emotion recognition result comprises: Input the multimodal emotion feature vector into the deep neural network to obtain the probability distribution of various emotions; According to the probability distribution of various emotions and combined with the attention mechanism, the weights of different modalities are dynamically adjusted to give the final emotion recognition result.

6. A multimodal XR emotional interaction system, characterized in that: The system comprises: The data collection module collects the user's multimodal data through the multimodal perception interface of the XR device; The emotion recognition module pre-processes the collected multimodal data, uses a deep neural network to perform emotion recognition on the pre-processed multimodal data, and obtains the emotion recognition results; The emotion understanding module builds an emotion knowledge base, maps the emotion recognition results into the emotion knowledge base, and performs reasoning and analysis on the current emotion; The emotion regulation module automatically classifies emotions according to the user's personalized emotion model, and intervenes and regulates the user's emotions based on the user's current emotions, personalized emotion model and emotion regulation reinforcement learning algorithm.

7. The multimodal XR emotion interaction system according to claim 6, characterized in that: The data acquisition module comprises: A camera to capture the user's facial expressions; Microphone, used to collect user's voice signal; Tracking sensors, including head tracking sensors and hand tracking sensors, for collecting the user's head posture and gesture movements; Eye tracking module, used to collect the user's gaze point and line of sight movement trajectory; Wearable sensors are used to collect physiological signals from users.

8. The multimodal XR emotion interaction system according to claim 6, characterized in that: The emotion recognition module is also used for: Convert multimodal data into a unified digital form; Perform noise reduction, smoothing and normalization processing on the converted multimodal data; Align the timestamps of the processed multimodal data and extract key features that can reflect the emotional state; The key features are fused according to a predefined data structure to form a unified multimodal emotion feature vector.

9. A computer device comprising a memory and a processor, wherein the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the multimodal XR emotion interaction method according to any one of claims 1 to 5.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the multimodal XR emotion interaction method according to any one of claims 1 to 5.

Images