Multi-mode VR interaction system and method for depression rehabilitation

Abstract

The invention discloses a multi-mode VR interaction system and method for depression rehabilitation, and the system comprises a signal collection device which is used for synchronously collecting physiological signals of a user; the reinforcement learning device is used for determining scene adjustment parameters based on the user physiological signals; the intelligent interaction device is used for adjusting at least one of the visual sense, the touch sense and the smell sense of the user based on the scene adjustment parameters, and the reinforcement learning device comprises a state recognition unit; a strategy optimization unit; and a scene parameter adjustment unit. According to the system, accurate monitoring and dynamic regulation and control of the physiological state of a patient are achieved, the rehabilitation effect is improved, ruminant thinking is reduced, a data-driven mediation effect model is constructed, and a brand-new and all-around technical solution is provided for depression positive breathing rehabilitation.

Description

Technical Field

[0001] This application relates to the field of VR interaction technology, and particularly to a multi-modal VR interaction system and method for depression rehabilitation. Background Art

[0002] The non-pharmacological intervention technology for depression is a current research hotspot. Virtual reality (VR) combined with biofeedback technology is gradually applied to the field of psychological rehabilitation due to its immersive experience and real-time regulation ability. VR technology is gradually applied to the field of psychological rehabilitation due to its immersive experience and real-time regulation ability. It can create a highly immersive and controllable virtual environment, enabling patients to conduct psychological intervention through "experience" rather than "imagination". Its core lies in allowing patients to perform safe exposure, situation reproduction, and relaxation training in the virtual environment. Existing VR rehabilitation systems mostly adopt single-modal signals (such as EEG or HRV), lacking the ability of multi-modal data fusion, resulting in lagging scene adaptation.

[0003] Therefore, the current VR rehabilitation systems still need to be improved. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art to some extent. For this purpose, this application provides a multi-modal VR interaction system and method for depression rehabilitation. Through technical means such as high-precision multi-modal signal synchronization, personalized scene parameter regulation, multi-modal stimulus mapping, efficient data processing and feature extraction, closed-loop regulation optimization, cognitive load compensation and personalized adaptation, and comprehensive physiological state monitoring and safety protection, the system realizes the precise monitoring and dynamic regulation of the patient's physiological state, improves the rehabilitation effect and reduces rumination thinking, constructs a data-driven mediating effect model, and provides a new and comprehensive technical solution for mindfulness breathing rehabilitation of depression.

[0005] In the first aspect of this application, a multi-modal VR interaction system is proposed. According to an embodiment of this application, the system includes: a signal collection device for synchronously collecting the physiological signals of a user; a reinforcement learning device for determining scene adjustment parameters based on the user's physiological signals; and an intelligent interaction device for adjusting at least one of the user's vision, touch, and smell based on the scene adjustment parameters. The reinforcement learning device includes: a state recognition unit for determining the physiological state of the user based on the physiological signals; a policy optimization unit for determining a user state optimization strategy based on the physiological state and the current scene; and a scene parameter adjustment unit for determining the scene adjustment parameters based on the user state optimization strategy and determining whether the user optimization strategy matches the user.

[0006] Thus, the physiological signals of the user are synchronously acquired by the signal collection device as the basic data for subsequent regulation. The reinforcement learning device receives these signals. Its state recognition unit first determines the current physiological state of the user, and then the policy optimization unit formulates the optimal user state optimization policy in combination with the current scenario. Finally, the scenario parameter adjustment unit determines the specific scenario adjustment parameters according to this policy. The intelligent interaction device precisely adjusts the sensory experiences of the user such as vision, touch, and smell according to these parameters, forming a closed loop from signal collection, state analysis to scenario optimization, so as to realize the dynamic monitoring and personalized regulation of the user's physiological state, and improve the rehabilitation effect and user experience.

[0007] According to an embodiment of the present application, the physiological signals include at least one of electroencephalogram signals, electrocardiogram signals, and respiratory signals.

[0008] According to an embodiment of the present application, the physiological state is extracted by a multi-modal spatio-temporal graph network.

[0009] According to an embodiment of the present application, the optimization strategy unit is optimized by using a multi-objective deep deterministic policy gradient algorithm.

[0010] According to an embodiment of the present application, the user state optimization policy includes an expected ECG R-R interval sequence.

[0011] According to an embodiment of the present application, the scenario parameter adjustment unit includes: an STCN neural network, which receives the expected ECG R-R interval sequence and outputs the physiological compatibility between the expected ECG R-R interval sequence and the user and the scenario adjustment parameters for realizing the expected ECG R-R interval sequence.

[0012] According to an embodiment of the present application, the intelligent interaction device further includes: a visual channel; an auditory channel; a tactile channel; a real-time rendering optimization module; a user behavior feedback module.

[0013] In the second aspect of the present application, a method for multimodal VR interaction is proposed. According to an embodiment of the present application, it includes: signal collection, which is used to synchronously collect the physiological signals of the user; reinforcement learning, which is used to determine scene adjustment parameters based on the user's physiological signals; intelligent interaction, which is used to adjust at least one of the user's vision, touch, and smell based on the scene adjustment parameters, where the reinforcement learning includes: a state recognition unit, which is used to determine the physiological state of the user based on the physiological signals; a policy optimization unit, which is used to determine a user state optimization policy based on the physiological state and the current scene; and a scene parameter adjustment unit, which is used to determine the scene adjustment parameters based on the user state optimization policy and determine whether the user optimization policy matches the user.

[0014] In the third aspect of the present application, an electronic device is proposed. According to an embodiment of the present application, it includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the system described in the first aspect is implemented.

[0015] In the fourth aspect of the present application, a readable storage medium is proposed. According to an embodiment of the present application, the readable storage medium stores computer-readable instructions, and the computer-readable instructions are used to cause a computer to execute the system described in the first aspect.

[0016] The additional aspects and advantages of the present application will be partly given in the following description, partly will become obvious from the following description, or will be understood through the practice of the present application. Description of the Drawings

[0017] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where:

[0018] Figure 1 It is a schematic diagram of the VR interaction system in Embodiment 1 of the present application. Detailed Embodiments

[0019] The embodiments of the present application will be described in detail below. The following described embodiments are exemplary and are only used to explain the present application, and should not be construed as a limitation to the present application.

[0020] It should be noted that the terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. Further, in the description of the present application, unless otherwise stated, "a plurality of" means two or more.

[0021] The endpoints and any values disclosed in the ranges herein are not limited to the exact ranges or values. These ranges or values should be understood to include values close to these ranges or values. For numerical ranges, between the endpoint values of each range, between the endpoint values of each range and individual point values, and between individual point values, they can be combined with each other to obtain one or more new numerical ranges, and these numerical ranges should be regarded as specifically disclosed herein.

[0022] In this document, the term "comprising" or "including" is an open-ended expression, that is, it includes the content specified in the present application, but does not exclude other aspects of the content.

[0023] In this document, the terms "optionally", "optional" or "option" generally mean that the subsequent events or conditions may or may not occur, and this description includes the cases where the events or conditions occur and the cases where the events or conditions do not occur.

[0024] Terms and Definitions

[0025] In this document, the term "EEG signal" refers to an electroencephalogram signal.

[0026] In this document, the term "ECG" refers to an electrocardiogram signal.

[0027] In this document, the term "ECG R-R interval sequence" refers to the time interval sequence between consecutive R waves in an electrocardiogram (ECG) signal. The R wave is a component of the QRS complex in the ECG signal, usually representing the beginning of ventricular depolarization and being one of the most prominent peaks in the ECG signal. The R-R interval sequence reflects the rhythm and time interval changes of the heartbeat and is an important indicator for analyzing heart rate variability (HRV) and cardiac autonomic nervous system activity.

[0028] In this document, the term "STCN neural network" is a deep learning model that combines a temporal convolutional network (TCN) and an attention mechanism.

[0029] In this document, the term "EOA", that is, electrocardiogram oscillation asymmetry, is a characteristic index for analyzing electrocardiogram signals, and it reflects the electrophysiological characteristics of the heart and the regulatory state of the autonomic nervous system by evaluating the asymmetry in the electrocardiogram signal.

[0030] Multimodal VR Interaction System

[0031] In the first aspect of the present application, a multimodal VR interaction system is proposed. According to an embodiment of the present application, the system includes: a signal collection device for synchronously collecting physiological signals of a user; a reinforcement learning device for determining scene adjustment parameters based on the user's physiological signals; and an intelligent interaction device for adjusting at least one of the user's vision, touch, and smell based on the scene adjustment parameters, wherein the reinforcement learning device includes: a state recognition unit for determining the physiological state of the user based on the physiological signals; a policy optimization unit for determining a user state optimization strategy based on the physiological state and the current scene; and a scene parameter adjustment unit for determining the scene adjustment parameters based on the user state optimization strategy and determining whether the user optimization strategy matches the user.

[0032] Thus, the signal collection device synchronously obtains the physiological signals of the user as the basic data for subsequent regulation; the reinforcement learning device receives these signals, its state recognition unit first determines the current physiological state of the user, then the policy optimization unit combines the current scene to formulate the optimal user state optimization strategy, and finally the scene parameter adjustment unit determines the specific scene adjustment parameters according to this strategy; the intelligent interaction device adjusts the sensory experiences such as the user's vision, touch, and smell accurately according to these parameters, forming a closed loop from signal collection, state analysis to scene optimization, so as to realize the dynamic monitoring and personalized regulation of the user's physiological state, and improve the rehabilitation effect and user experience.

[0033] According to an embodiment of the present application, the physiological signals include at least one of electroencephalogram signals, electrocardiogram signals, and respiratory signals. Thus, electroencephalogram signals are used to monitor the electrical activity of the brain and reflect the user's neural activity state; electrocardiogram signals are used to monitor the electrical activity of the heart and reflect the user's cardiovascular state; respiratory signals are used to monitor the user's breathing rhythm and depth and reflect the user's respiratory state. By collecting these physiological signals, the system can comprehensively understand the user's physiological condition and provide accurate data support for subsequent physiological state evaluation and scene parameter adjustment.

[0034] According to an embodiment of the present application, the physiological state is extracted by a multimodal spatio-temporal graph network. Thus, the multimodal spatio-temporal graph network can integrate various physiological signals, such as electroencephalogram (EEG), electrocardiogram (ECG), respiratory signals, etc., capture the complex relationships of these signals in space and time, and thus extract and analyze the user's physiological state more comprehensively and accurately.

[0035] According to some specific embodiments of the present application, an FPGA hardware clock synchronization module is adopted to achieve cross-modal alignment of EEG, ECG, and PPG. Thereby, it ensures that physiological signals of different modalities are accurately synchronized in time, and the synchronization error is controlled at an extremely low level, providing guarantee for the fusion and analysis of multi-modal data.

[0036] According to some specific embodiments of the present application, for the microstate features in EEG signals, an improved hidden Markov model is adopted for real-time recognition, which can accurately distinguish between class C and class D microstates. Microstate is a concept in electroencephalogram (EEG) signal analysis, referring to the stable distribution pattern of cerebral cortical electrical activities in a short period of time (usually from dozens of milliseconds to hundreds of milliseconds). Thereby, it provides key neural state information for subsequent decision-making.

[0037] According to some specific embodiments of the present application, by calculating the multi-scale entropy (MSE) of heart rate variability (HRV) and the coefficient of variation of pulse wave transit time (CV-PTT) of photoplethysmogram (PPG), the system can comprehensively evaluate the cardiovascular health and autonomic nervous system activities of the user. The multi-scale entropy of HRV reflects the complexity and regulation ability of the cardiac autonomic nervous system by calculating sample entropy at different time scales, while the coefficient of variation of pulse wave transit time of PPG evaluates the fluctuation of pulse wave transit time by calculating the coefficient of variation of the PTT sequence. Combining these two analysis methods, the system can more accurately monitor the physiological state of the user, and accordingly dynamically adjust the VR scene parameters to optimize the rehabilitation training effect and provide a personalized rehabilitation plan for the user.

[0038] According to the embodiments of the present application, the optimization strategy unit is optimized by using the multi-objective deep deterministic policy gradient algorithm. Thereby, it dynamically adjusts parameters such as vision, touch, and smell, so as to enhance the rehabilitation effect.

[0039] According to the embodiments of the present application, the user state optimization strategy includes the desired ECG R-R interval sequence. Thereby, the ECG R-R interval sequence reflects the rhythm of the heart and the activity state of the autonomic nervous system, and is an important indicator for measuring the physiological state of the user. It can dynamically adjust scene parameters, such as the intensity and frequency of visual, tactile, and olfactory stimuli, to guide the physiological state of the user to change in the desired direction, achieve precise regulation of the user's autonomic nervous system, improve the rehabilitation effect, reduce rumination, and enhance the relaxation and treatment effect of the user in the VR environment.

[0040] According to some specific embodiments of the present application, a dynamic light sphere in Unity 3D (diameter change rate ΔD / Δt = 0.5 - 1.2 cm / s) is used to match the breathing depth, and its diameter change rate is matched with the patient's breathing depth to guide the patient to perform rhythmic breathing training. The color of the light sphere adopts the HSV color space to map the α power (α < 8 μV2 → cold color tone, α > 15 μV2 → warm color tone): According to the real-time calculated α power value, the color of the light sphere is dynamically switched between the cold color tone and the warm color tone in the HSV color space to provide visual feedback to the patient and assist them in adjusting the breathing rhythm. The phase response curve (PRC) is used to optimize the phase-locking efficiency of visual stimuli on the breathing rhythm: By experimentally measuring the response of the breathing rhythm to visual stimuli at different phases, a PRC curve is constructed, and according to this curve, the visual stimulus parameters of the light sphere are optimized to improve the phase-locking efficiency of visual stimuli on the breathing rhythm.

[0041] According to some specific embodiments of the present application, the reinforcement learning decision-making layer includes: a high-level state recognition layer; a middle-level policy optimization layer; and a low-level temporal adaptation layer.

[0042] According to some specific embodiments of the present application, the high-level state recognition layer analyzes the EEG microstate transition matrix through a multi-modal spatio-temporal graph network (MT-GCN) to output the suppression level (level 0 - 4) of the default mode network (DMN), thereby enhancing physiological interpretability. At the same time, this network also integrates the vagal tone index (HF-HRV > 0.4) in the ECG signal to enhance the comprehensive assessment of the patient's physiological state. Specifically, MT-GCN can capture the dynamic conversion patterns between EEG microstates and judge the degree of DMN suppression by analyzing these patterns. The higher the DMN suppression level, the higher the patient's brain concentration in a specific cognitive task. The vagal tone index (HF-HRV > 0.4) is an index used to evaluate the level of vagal nerve activity by analyzing the high-frequency component in heart rate variability (HRV). An increase in the high-frequency component is usually associated with an enhancement of vagal tone, which reflects the balance state of the autonomic nervous system. By combining the analysis of these two multi-modal signals, the system can more comprehensively evaluate the patient's physiological state and provide a more accurate basis for subsequent personalized rehabilitation training.

[0043] According to some specific embodiments of the present application, the middle layer of the policy optimization is optimized and adjusted using the Multi-Objective Deep Deterministic Policy Gradient (MO-DDPG) algorithm. The MO-DDPG algorithm is a reinforcement learning algorithm specifically designed to handle multi-objective optimization problems and can find the optimal policy balance in a complex decision space. In the present application, this algorithm is optimized for two key objectives: the neuromodulation objective and user safety. The neuromodulation objective refers to optimizing the user's neural state by adjusting VR scene parameters, such as adjusting the suppression level of the default mode network (DMN); user safety involves ensuring that the modulation strategy does not have an adverse impact on the user, such as avoiding physiological discomfort caused by excessive stimulation. Through the MO-DDPG algorithm, the system can ensure the safety of the user while pursuing the neuromodulation effect, thus realizing an effective and safe personalized rehabilitation strategy.

[0044] According to some specific embodiments of the present application, the input of the STCN (Spatio-Temporal Convolutional Network) network in the underlying temporal adaptation is adjusted to the ECG R-R interval sequence for analyzing the time series characteristics of heart rate variability (HRV). In this way, the STCN network can dynamically adjust the VR scene parameters according to the user's cardiac rhythm.

[0045] According to an embodiment of the present application, the scene parameter adjustment unit includes: an STCN neural network, which receives the desired ECG R-R interval sequence and outputs the physiological compatibility between the desired ECG R-R interval sequence and the user and the scene adjustment parameters for realizing the desired ECG R-R interval sequence. Thereby, it is used to guide the intelligent interaction device to adjust the sensory stimuli such as vision, touch, and smell of the user to guide the physiological state of the user to change in the desired direction, realize the precise regulation of the user's autonomic nervous system, and improve the rehabilitation effect and user experience.

[0046] According to an embodiment of the present application, the intelligent interaction device further includes: a visual channel; an auditory channel; a tactile channel; a real-time rendering optimization module; and a user behavior feedback module. Thus, the visual channel uses the Unity HDRP engine to dynamically adjust the scene complexity, map the γ oscillation intensity, and dynamically adjust the ambient color temperature according to the ECG-PPG coupling index; the auditory channel realizes real-time synchronization between the binaural beat frequency and the hippocampal θ phase; the tactile channel jointly controls the tactile feedback intensity according to the α power and EOA. The real-time rendering optimization module uses technologies such as the DDS protocol to achieve adaptive rendering resolution, ensuring the smoothness and high-quality rendering of the VR scene. The user behavior feedback module monitors the coefficient of variation of the user's pupil diameter and EOA to determine whether the user has excessive sympathetic activation, and triggers a downgrade in scene complexity. At the same time, it integrates the Whisper voice model to identify the user's emergency interruption command. These modules work together to adjust parameters according to the scene, accurately adjust the user's visual, auditory, and tactile sensory experiences, forming a closed loop from signal collection, status analysis to scene optimization, so as to achieve dynamic monitoring and personalized regulation of the user's physiological state, improving the rehabilitation effect and user experience.

[0047] According to some specific embodiments of the present application, the intelligent interaction device further includes: an abort module; the abort module determines whether the user has a state of excessive sympathetic activation by real-time monitoring of the coefficient of variation of the user's pupil diameter (when it exceeds 25%) and electrocardiogram oscillation asymmetry (EOA, when it is greater than 1.5). Once it is detected that the above conditions are met, the system will automatically trigger a downgrade in the VR scene complexity to reduce the user's physiological load and prevent adverse effects caused by excessive stimulation to the user. At the same time, this module integrates an advanced Whisper voice model, which can accurately identify the user's emergency interruption command (such as "stop"), ensuring that the user can terminate the training in a timely manner in any uncomfortable situation, further improving the safety and user experience of the system.

[0048] In a second aspect of the present application, the present application proposes a method for multi-modal VR interaction. According to an embodiment of the present application, it includes: signal collection, which is used to synchronously collect the user's physiological signals; reinforcement learning, which is used to determine scene adjustment parameters based on the user's physiological signals; intelligent interaction, which is used to adjust at least one of the user's vision, touch, and smell based on the scene adjustment parameters, wherein the reinforcement learning includes: a state recognition unit, which is used to determine the user's physiological state based on the physiological signals; a policy optimization unit, which is used to determine a user state optimization policy based on the physiological state and the current scene; and a scene parameter adjustment unit, which is used to determine the scene adjustment parameters based on the user state optimization policy and determine whether the user optimization policy matches the user.

[0049] Those skilled in the art can understand that the features and advantages described above for the multi-modal VR interaction system also apply to this method, and will not be elaborated here.

[0050] In the third aspect of the present application, the present application proposes an electronic device. According to an embodiment of the present application, it includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the system described in the first aspect is implemented.

[0051] Those skilled in the art can understand that the features and advantages described above for the multi-modal VR interaction system also apply to this electronic device, and will not be elaborated here.

[0052] In the fourth aspect of the present application, the present application proposes a readable storage medium. According to an embodiment of the present application, the readable storage medium stores computer-readable instructions for causing a computer to execute the system described in the first aspect.

[0053] Those skilled in the art can understand that the features and advantages described above for the multi-modal VR interaction system also apply to this readable storage medium, and will not be elaborated here.

[0054] The solution of the present application will be explained below in conjunction with embodiments. Those skilled in the art will understand that the following embodiments are only used to illustrate the present application and should not be regarded as limiting the scope of the present application. For those not specified in the embodiments regarding specific technologies or conditions, they shall be carried out according to the technologies or conditions described in the literature in the art or according to the product specifications. For reagents or instruments not specified by the manufacturer, they are all conventional products that can be obtained through commercial purchases.

[0055] Embodiment 1: Basic Parameter Setting and System Verification

[0056] The VR interaction system of the present application is as Figure 1 shown.

[0057] Hardware configuration: Multi-modal sensors: Embedded EEG electrode patches (three-lead forehead), flexible breathing belts (piezoelectric), and ECG patches (single-lead) are adopted.

[0058] Synchronization module: A hardware clock synchronization circuit based on FPGA (Xilinx Artix-7 series) is used to ensure the time alignment of EEG, ECG, and breathing signals, and the synchronization error is controlled within 5 ms.

[0059] Data Processing and Decision-making: Feature Extraction: Extract the frontal alpha asymmetry (FAA) from EEG signals and the low-frequency / high-frequency power ratio (LF / HF) from ECG signals. Reinforcement Learning Decision-making: Construct a state space S = {FAA, LF / HF, respiratory rate} and an action space A = {visual complexity, background music rhythm, tactile feedback intensity}, and use the Q-learning algorithm to optimize the mapping of scenario parameters.

[0060] In the Q-learning algorithm, the learning rate α is 0.1 and the discount factor γ is 0.9.

[0061] Set the diameter change rate (ΔD / Δt) of the dynamic light sphere in Unity 3D to 0.5 - 1.2 cm / s; the frequency range of binaural beats is 40 - 60 Hz; the adjustment range of tactile feedback intensity is 0.1 - 1.0 N.

[0062] Intelligent Interaction: Convert the decision-making instructions into perceivable interaction parameters to achieve closed-loop regulation of neural states and scenario parameters.

[0063] Example 2: Reinforcement Learning Parameter Adjustment

[0064] Perform reinforcement learning parameter adjustment on the VR interaction system in Example 1.

[0065] Reinforcement Learning Parameters: Adjust the learning rate α and the discount factor γ in the Q-learning algorithm, adjust α from 0.1 to 0.3, and γ from 0.9 to 0.8.

[0066] Example 3: Optimization of Multimodal Stimulation Parameters

[0067] Perform optimization of multimodal stimulation parameters on the VR interaction system in Example 1.

[0068] Visual Stimulation: Change the diameter change rate (ΔD / Δt) of the dynamic light sphere in Unity 3D, adjusted from 0.5 - 1.2 cm / s to 1.0 - 1.5 cm / s.

[0069] Auditory Stimulation: Adjust the frequency range of binaural beats, adjusted from 40 - 60 Hz to 50 - 70 Hz.

[0070] Tactile Stimulation: Change the adjustment range of tactile feedback intensity, adjusted from 0.1 - 1.0 N to 0.2 - 1.2 N.

[0071] Test Example

[0072] Four groups of depression patients were selected, with 20 patients in each group. Among them, three groups were respectively given mindfulness breathing training using the systems of Examples 1 to 3 for 4 weeks, and a comparative example was set up for ordinary mindfulness breathing training for 4 weeks. The physiological indicators and scores of the psychological assessment scale of the patients were recorded. The average score results of the Beck Depression Inventory (BDI) for each group are shown in Table 1. It is proved that the multi-modal VR interaction system of the present application can effectively relieve the depressive symptoms of patients. After adjusting the reinforcement learning parameters, the rehabilitation effect of patients can be further improved. By optimizing the multi-modal stimulation parameters, the rehabilitation effect can be further improved.

[0073] Table 1. Psychological Assessment Scale

[0074] Sample Average BDI Score (points) Example 1 17 Example 2 12 Example 3 6 Comparative Example 24

[0075] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0076] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A multimodal VR interaction system, characterized in that, Comprising: A signal collection device for synchronously collecting physiological signals of a user; A reinforcement learning device for determining scene adjustment parameters based on the user's physiological signals; An intelligent interaction device for adjusting at least one of the user's vision, touch, and smell based on the scene adjustment parameters, wherein, The reinforcement learning device includes: A state recognition unit for determining the physiological state of the user based on the physiological signals; A policy optimization unit for determining a user state optimization policy based on the physiological state and the current scene; and A scene parameter adjustment unit for determining the scene adjustment parameters based on the user state optimization policy and determining whether the user optimization policy matches the user.

2. The system according to claim 1, characterized in that, The physiological signals include at least one of electroencephalogram signals, electrocardiogram signals, and respiratory signals.

3. The system according to claim 1, wherein The physiological state is extracted by a multi-modal spatio-temporal graph network.

4. The system according to claim 1, characterized in that, The optimization policy unit is optimized using a multi-objective deep deterministic policy gradient algorithm.

5. The system according to claim 1, wherein The user state optimization policy includes an expected ECG R-R interval sequence.

6. The system according to claim 5, characterized in that, The scene parameter adjustment unit includes: An STCN neural network that receives the expected ECG R-R interval sequence and outputs the physiological compatibility of the expected ECG R-R interval sequence with the user and the scene adjustment parameters for achieving the expected ECG R-R interval sequence.

7. The system according to claim 1, wherein The intelligent interaction device further includes: A visual channel; An auditory channel; A tactile channel; A real-time rendering optimization module; A user behavior feedback module.

8. A method for multimodal VR interaction, characterized in that, Comprising: Signal collection for synchronously collecting physiological signals of a user; Reinforcement learning for determining scene adjustment parameters based on the user's physiological signals; Intelligent interaction for adjusting at least one of the user's vision, touch, and smell based on the scene adjustment parameters, wherein, The reinforcement learning includes: A state recognition unit for determining the physiological state of the user based on the physiological signals; A policy optimization unit for determining a user state optimization policy based on the physiological state and the current scene; and A scene parameter adjustment unit for determining the scene adjustment parameters based on the user state optimization policy and determining whether the user optimization policy matches the user.

9. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the system according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores computer-readable instructions for causing a computer to execute the system according to any one of claims 1 to 7.

Images