Multi-modal data fused virtual reality assisted diagnosis and treatment method for depression

Abstract

The invention provides a depression virtual reality auxiliary diagnosis and treatment method fused with multi-modal data, and the method comprises the steps: obtaining the multi-modal data, the multi-modal data comprises the peripheral physiological data, such as heart rate, body temperature and skin electricity, monitored by a flexible sensor based on epidermal electrons; and switching to a personalized interactive virtual scene according to the emotional state of the patient in real time, and in the interaction process, obtaining eyeball movement data by the eye movement tracking instrument, and knowing attention distribution, psychological state and micro facial expression change of the user. In combination with multi-modal data of peripheral physiological data, eye movement data and micro-expression change, key features are automatically extracted from multi-modal input data, and emotion analysis and recognition are performed through a multi-modal deep learning method with an attention mechanism. Data can be collected and analyzed in real time, the emotional state of a patient is objectively reflected through the combination of peripheral physiological data and micro-expression analysis, the requirements of the patient are met in a personalized mode, and the treatment effect is improved.

Description

Technical Field

[0001] The present invention belongs to the cross - technical field of medicine and computer, and particularly relates to a virtual reality assisted diagnosis and treatment method for depression integrating multi - modal data. Background Art

[0002] The traditional diagnosis and treatment method of depression is a complex and multi - dimensional process, which involves detailed inquiries about the patient's self - reported symptoms, doctors' clinical observations, and the application of standardized psychological assessment tools, etc. This process starts with a clinical interview, where doctors collect information about the patient's emotions, behaviors, thinking patterns, and daily life functions through conversations with the patient. This face - to - face communication is crucial for understanding the patient's mental state and their subjective observations of symptoms. However, the existing medical resources are insufficient to meet the needs of a large number of emerging patients, and automated assisted diagnosis and treatment methods are required.

[0003] Existing depression diagnosis and treatment devices usually only utilize a single type of data, for example, only evaluate through peripheral physiological data (such as heart rate, temperature, and skin conductance response, etc.) or only through facial expressions. In the related patents of the facial expression detection and treatment system proposed by Akili Interactive LabsInc., a system that uses facial expression recognition and simulation workflows to train users' cognitive and emotional empathy is described. By prompting users to imitate specific emotional expressions and capturing the users' facial expressions to evaluate their emotional states. Although this method is effective, it mainly relies on facial expression data and ignores the influence of other physiological data, resulting in the problem of data singularity and being unable to comprehensively reflect the patient's mental state. Many existing devices are unable to process and analyze data in real - time, making it difficult to provide timely feedback and affecting the diagnosis and treatment efficiency. Summary of the Invention

[0004] To solve the above - mentioned technical problems, the present invention proposes a personalized virtual reality assisted diagnosis and treatment method based on multi - modal data fusion. By real - time analyzing the patient's peripheral physiological data, eye movement data, and micro - expression data, the scene settings in the virtual reality environment are dynamically adjusted, thereby realizing personalized emotion management and psychotherapy. Specifically, while the patient is watching the virtual reality scene, the system can adjust the lighting, background music, interactive elements, etc. in the virtual scene according to the patient's real - time emotional state. After obtaining enough data to get an accurate result of emotion recognition, it switches to the corresponding virtual reality scene for symptomatic emotion regulation and assisted treatment to provide a diagnosis and treatment experience that better meets the individual needs of the patient.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A virtual reality assisted diagnosis and treatment method for depression integrating multi - modal data, the method includes:

[0007] Step 1: Obtain peripheral physiological data through an epidermal electronic flexible sensor, obtain eye movement data through an eye tracker, and obtain micro-expression data through a high-precision camera;

[0008] Step 2: Preprocess and extract features from the peripheral physiological data, eye movement data, and micro-expression data to obtain multi-modal feature data;

[0009] Step 3: Use a multi-modal deep learning model based on the attention mechanism to perform fusion analysis on the multi-modal feature data to identify the patient's emotional state;

[0010] Step 4: Dynamically adjust the virtual reality scene according to the identified emotional state to provide personalized emotional regulation and adjuvant therapy.

[0011] Further, in Step 1, the peripheral physiological data includes the patient's heart rate, body temperature, and skin electrical activity data, the eye movement data includes the patient's fixation point position, fixation duration, eye movement trajectory, blink frequency, and pupil diameter change data, and the micro-expression data includes the patient's facial micro-expression change data.

[0012] Further, in Step 2, for the peripheral physiological data, use a Butterworth filter to remove high-frequency noise, and use a bidirectional long short-term memory network to process the filtered peripheral physiological data to extract peripheral physiological time features.

[0013] Further, in Step 2, for the eye movement data, use a Savitzky-Golay filter for correction and smoothing, use principal component analysis to reduce the dimension of the filtered eye movement data to obtain low-dimensional features, and use a feature extraction network based on a convolutional neural network to perform deep feature extraction on the low-dimensional features to obtain an eye movement comprehensive feature vector.

[0014] Further, in Step 2, for the micro-expression data, use the ASM algorithm to locate the key feature points on the face, and use the depth convolution and pointwise convolution of the DSCNNs network to process the extracted facial key feature point data to obtain facial fusion features.

[0015] Further, in Step 3, using the multi-modal deep learning model based on the attention mechanism to perform fusion analysis on the multi-modal feature data includes,

[0016] Taking the peripheral physiological time features, eye movement comprehensive feature vector, and facial fusion features as query vectors respectively, and obtaining the cross-attention matrices corresponding to the peripheral physiological time features, eye movement comprehensive feature vector, and facial fusion features respectively through cross-modal attention weight calculation;

[0017] Based on the respective corresponding cross-attention matrices, a linear transformation layer is used to adaptively fuse the peripheral physiological time features, the comprehensive eye movement feature vector, and the facial fusion features;

[0018] A residual connection structure is introduced, and the features after adaptive fusion are superimposed with the peripheral physiological time features, the comprehensive eye movement feature vector, and the facial fusion features;

[0019] Through feature splicing operations, the superimposed features are hierarchically aggregated to construct a multi-modal feature space with rich semantic representations.

[0020] Further, in step 3, identifying the emotional state of the patient includes identifying the emotion category and emotion intensity. The emotion category identification is based on a Softmax function to construct an emotional category probability distribution generator, and the output layer generates a five-dimensional probability vector including pleasure, anger, disgust, fear, and sadness. The confidence value of each dimension represents the possibility intensity of the corresponding emotion category; the emotion intensity assessment uses a two-dimensional orthogonal emotion space composed of valence and arousal value to quantify the emotion intensity. Among them, valence represents the positive and negative polarity of the emotion, and arousal value represents the physiological activation degree of the emotion.

[0021] Further, step 4 includes using a sliding time window algorithm to verify the convergence of the emotional state. When the system continuously obtains high-confidence emotion classification results within a preset time window, it switches to the corresponding VR scene according to the classification results and starts a preset emotion regulation plan; if it detects significant fluctuations in the emotional state, it executes a window extension strategy and implements continuous monitoring until the emotion steady-state criterion is met.

[0022] In a second aspect, the present invention provides an electronic device, including: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the foregoing personalized virtual reality-assisted diagnosis and treatment method based on multi-modal data fusion.

[0023] In a third aspect, the present invention provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is enabled to implement the foregoing personalized virtual reality-assisted diagnosis and treatment method based on multi-modal data fusion.

[0024] The beneficial effects of the present invention are as follows:

[0025] Compared with a single data source, the present invention adopts a multimodal data fusion method to fuse and analyze the data obtained from epidermal electronic flexible sensors, eye trackers, and video micro-expression analysis technologies, so as to obtain the emotional state of patients and adjust it. Multimodal data can more comprehensively and accurately reflect the emotional and physiological states of patients, reducing the possibility of misdiagnosis and missed diagnosis. By using virtual reality technology, an immersive diagnosis and treatment environment is created, allowing patients to perform scenario simulations and interactions in natural interactions, increasing the realism of the experience and the effectiveness of treatment. At the same time, the virtual reality environment can shield external interference, making data collection more stable and accurate.

[0026] The present invention performs emotion recognition through a multimodal deep learning method with an attention mechanism, which can automatically extract key features from multimodal input data, thereby improving the recognition accuracy.

[0027] The present invention realizes real-time feedback and adjustment of the collected multimodal data, can capture and analyze physiological and micro-expression data in real time, provide instant feedback, and helps doctors and patients quickly understand the current emotional state and physiological reactions of patients. According to the real-time data feedback, the present invention can dynamically adjust the treatment plan to meet the needs of patients personalized and improve the treatment effect.

[0028] The present invention can automatically adjust treatment parameters and plans according to the specific conditions of different patients, providing personalized treatment services. At the same time, it can also record and analyze the long-term data change trends of patients, helping doctors better understand and predict the emotional and physiological states of patients and optimize the treatment plan. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of a virtual reality-assisted diagnosis and treatment method for depression that integrates multimodal data according to the present invention;

[0030] Figure 2 is a network schematic diagram of a virtual reality-assisted diagnosis and treatment method for depression that integrates multimodal data according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] The present invention will be further described below with reference to the drawings and embodiments.

[0032] The present invention proposes a virtual reality-assisted diagnosis and treatment method for depression that integrates multimodal data. By combining peripheral physiological data, eye movement data, micro-expression data, and virtual reality technology, it aims to assist in solving problems such as lack of objective data, poor treatment compliance, and insufficient psychological treatment resources in the diagnosis and treatment process of depression. This method is specifically implemented through the following technical processes:

[0033] Obtaining peripheral physiological data;

[0034] The present invention adopts a high-performance epidermal electronic flexible sensor for accurately measuring peripheral physiological data. This flexible sensor is made of an organic polymer material with high conductivity and high stretchability, and can conform to the human skin to achieve continuous and stable data acquisition.

[0035] The epidermal electronic flexible sensor includes a conductive electrode and a flexible substrate. The electrode is patterned and deposited on the flexible substrate through micro-nano manufacturing processes. The epidermal electronic flexible sensor for obtaining peripheral physiological data has the following technical characteristics:

[0036] 1) High-conductivity and high-stretchability materials: A composite material of conductive polymer and silver nanowires is adopted to ensure that the flexible sensor maintains stable electrical performance under various stretching and bending states.

[0037] 2) The flexible sensor selects a thin-film structure design: The thickness of the flexible sensor is only dozens of micrometers, which can closely conform to the skin and reduce the foreign body sensation and discomfort during wearing.

[0038] 3) Multi-point distributed flexible sensor array: Multi-point flexible sensors are arranged at specific positions on the inner side of the helmet and the handle, covering the detection of physiological parameters such as electrocardiogram signal (ECG), electrodermal activity (EDA), and skin temperature (ST).

[0039] 4) Wireless data transmission module: Low-power Bluetooth or other wireless transmission technologies are adopted to ensure real-time and stable transmission of data to the processing unit, reducing the constraints and interference of wired connections.

[0040] 5) High-precision signal acquisition: A high-sensitivity amplifier and filter circuit are integrated to enhance the acquisition and anti-interference ability of weak physiological signals.

[0041] In terms of manufacturing process, the flexible sensor adopts precise photolithography and inkjet printing technologies to pattern and deposit the conductive material on the flexible substrate. A biocompatible polymer layer is coated on the surface of the final flexible sensor to avoid skin irritation and allergic reactions caused by long-term wearing.

[0042] Obtaining eye movement data;

[0043] The application of eye movement tracking technology in the diagnosis and treatment of psychological and mental diseases has gradually matured, especially in the auxiliary diagnosis and treatment of depression. The key data that an eye tracker can obtain include the position of the fixation point, fixation duration, eye movement trajectory, blink frequency, and pupil diameter change, etc. These data provide important bases for analyzing an individual's emotional state and cognitive process.

[0044] An eye tracker records the fixation positions of an individual under visual stimuli, accurate to the spatial coordinates of each frame. Depressed patients usually have a longer fixation time on negative emotional stimuli, showing a negative attention bias. By analyzing the fixation positions, the attention distribution of an individual to different emotional stimuli can be evaluated, thereby judging their emotional state.

[0045] The eye tracker measures the duration of an individual's fixation on a specific point. Depressed patients have a longer fixation duration on negative stimuli, and this characteristic can be used to identify and quantify emotional bias.

[0046] The eye tracker records the movement trajectories of the eyeballs in the horizontal and vertical directions, including the transitions between saccades and fixations. The eyeball movement trajectories of depressed patients are relatively rigid and lack flexibility, manifested as fewer saccades and longer fixation times. Analyzing these trajectories can evaluate an individual's information processing efficiency and emotional regulation ability.

[0047] The eye tracker records blink events and their frequencies. Blink frequency is closely related to psychological and emotional states. Depressed patients usually have a lower blink frequency, reflecting their difficulty in concentrating attention and low mood. Changes in blink frequency can provide real-time feedback on emotional state.

[0048] The eye tracker measures the diameter changes of the pupils under different lighting conditions and emotional stimuli. Pupil diameter change is an indicator of the activity of the autonomic nervous system. During the emotional arousal process of depressed patients, the pupil dilation response may be weaker. By monitoring the pupil diameter changes, the intensity of an individual's emotional response and arousal level can be evaluated.

[0049] Acquisition of micro-expression data;

[0050] For the acquisition of micro-expression data, a high-precision camera is required, and it needs to have a high resolution (at least 1080p or higher) to ensure the capture of subtle facial changes. A high frame rate (such as 120 frames per second or higher) is also crucial to record rapid micro-expression changes; the camera has excellent light-sensing ability to capture clear facial images under different lighting conditions, and color accuracy is also necessary for the correct analysis of skin tone changes (such as blushing); the camera can capture and transmit image data in real time, and high-speed data transmission capabilities and low-latency processing capabilities are required for real-time facial expression analysis. Use high-speed data transmission interfaces (such as USB 3.0 or above) and low-latency processing units to ensure real-time performance. Finally, the design of the camera needs to be compatible with virtual reality devices while remaining small and light enough. The integrated design should consider the comfort of the user and the overall weight of the device.

[0051] In summary, through the device of the present invention, physiological data of patients can be collected, visual attention and eye movement response data of patients can be obtained, and micro-expression data can be captured. These data can reflect the physiological responses and emotional states of patients, and can also reveal the attention allocation and reaction speed of patients to different stimuli. By analyzing subtle facial expression changes, it is also helpful to identify emotions that may not be expressed verbally.

[0052] After the collection of multi-modal data is completed, it is necessary to preprocess and extract features from the data, aiming to improve data quality, simplify the model, enhance model performance, and ensure data consistency. The specific methods are as follows:

[0053] Preprocessing and feature extraction for peripheral physiological data;

[0054] Preprocess the multi-modal peripheral physiological signals (including electrocardiogram signal ECG, electrodermal activity EDA, and skin temperature ST, etc.) collected based on epidermal electronic flexible sensors. First, use a Butterworth filter to filter and denoise the original data. The formula is as follows:

[0055] ,

[0056] where, is the filtered peripheral physiological data signal, is the cut-off frequency, is the time, is the instantaneous frequency of the original signal, is the filter order.

[0057] This method is mainly based on its characteristic of having the maximum flat amplitude response in the passband, which can effectively filter out high-frequency noise interference and retain the time-domain characteristics of the original physiological signal to the greatest extent. On this basis, according to the frequency-domain feature differences of different physiological signals, the filter parameters are adaptively adjusted to optimize the denoising effect of each signal and provide a high-quality data basis for subsequent feature extraction and analysis.

[0058] The bidirectional long short-term memory network (Bi-LSTM) architecture is often used to process time series data and can solve the problems of gradient explosion and information distortion. This network contains two different long short-term memory networks (LSTM), which obtain data from the forward and reverse directions respectively, and then integrate the information obtained from the two directions. The Bi-LSTM architecture can consider both the past and future information of data samples, taking into account the forward and reverse states of each sequence. The final output is obtained by merging the two hidden states. This method uses Bi-LSTM to extract features from the filtered peripheral physiological data, extracting the most relevant time features from the physiological signals to identify emotions. The formula is as follows:

[0059] ,

[0060] ,

[0061] ,

[0062] in, represents the maximum time step, is the forward LSTM at time step The hidden state of is the forward LSTM at time step The cell state is used to update the cell state and hidden state at the current moment. and is the reverse LSTM at time step The hidden state and cell state of . Indicates that Bi-LSTM is at time step The output vector of is the result of concatenating the forward and backward hidden vectors.

[0063] Preprocessing and feature extraction of eye movement data;

[0064] The raw eye movement data collected by the eye tracker (including gaze point coordinates, gaze duration, eye movement trajectory, blink frequency, pupil diameter change, etc.) are preprocessed. First, the Savitzky-Golay filter is used to correct and smooth the data. The formula is as follows:

[0065] ,

[0066] in, is the filtered eye movement data signal, is the original signal, is the relative offset within the time window, is the filter coefficient, is the window size.

[0067] This method effectively eliminates high-frequency noise in eye movement signals through local polynomial fitting, while retaining the time domain characteristics of the original data. On this basis, the data is further calibrated and compensated to eliminate system errors caused by factors such as head micro-movement and device offset, ensuring the accuracy of subsequent analysis.

[0068] Since direct eye movement features are shallow features with limited information and more noise, in order to mine deeper semantic relationships in the data, this method uses a feature extraction network based on convolutional neural networks (FE-CNN) to extract deeper features. This network can extract deep features without losing the original shallow features and fuse shallow features with deep features.

[0069] For the filtered eye movement data signal, principal component analysis (PCA) is used to reduce the dimension of the features, eliminating redundant and interfering information, such as the interference of light on the pupil diameter, etc. The formula is as follows:

[0070] ,

[0071] where, is the low-dimensional feature after dimension reduction, is the filtered eye movement data signal, is the result after centering the mean of the original data, is the projection matrix composed of the eigenvectors of the covariance matrix of the centered data.

[0072] The vector formed by the shallow features after PCA dimension reduction is input into the FE-CNN for the first deep feature extraction to extract the first-layer deep features . In order to retain the original information and supplement the abstract representation, the residual connection method is used to add and . This ensures that the original shallow features are retained, and at the same time, the features after the first-layer extraction are introduced. The fused features are input into the FE-CNN again for the second deep feature extraction to extract the second-layer deep features . Finally, through the concatenation operation, the original shallow features are concatenated with the deep features and extracted twice to form the comprehensive feature vector for subsequent feature fusion operations.

[0073] Preprocessing and feature extraction for micro-expression data;

[0074] For the original image data collected by the camera, face detection and facial feature localization are first performed. The present invention uses the Active Shape Model (ASM) algorithm to achieve accurate calibration of key facial feature points. This method establishes a statistical model of facial shape based on a large-scale labeled face dataset, learns the morphological change rules of facial structure, and adjusts the model shape parameters through an iterative optimization process to make the model feature points achieve the optimal match with the target facial features. This process is achieved by minimizing the energy function , and the energy function can be expressed as:

[0075] ,

[0076] where, is the weight corresponding to the th feature point, is the th feature point position observed, is the th feature point position predicted by the model, represents the feature point index.

[0077] Depthwise separable convolutional neural networks (DSCNNs) are popular deep learning architectures for image classification problems, with low memory consumption and low computational power requirements. The depthwise separable convolution operation decomposes the standard convolution into a depthwise convolution and a pointwise convolution. The key feature point data of the extracted face is processed using DSCNNs.

[0078] The depthwise convolution operation achieves minimal filtering by applying a filter to each input channel. Since each filter only computes its corresponding input channel, only in the spatial domain and without involving information fusion between different channels, the total amount of computation is reduced. The depthwise convolution operation can be represented by the following formula:

[0079] ,

[0080] where and represent the position coordinates of the output feature map in the horizontal and vertical directions respectively, and represent the positions of the convolutional kernel in the height and width directions respectively, represents the channel index of the input feature map, represents the depthwise convolutional kernel; represents the convolutional kernel weight for the th channel, with a size of ; represents the pixel value of the input feature map on the th channel at the spatial position , the size of the input feature map is usually , and represent the height and width respectively, is the number of input channels; represents the value of the th channel of the output feature map after depthwise convolution at the position .

[0081] The pointwise convolution operation aggregates the output of the depthwise convolution by applying convolutions. At each spatial position, a linear combination of all channels from the depthwise convolution is performed. It plays the role of cross-channel feature fusion, integrating the information of each channel to generate a more discriminative feature representation. The pointwise convolution operation can be represented by the following formula:

[0082] ,

[0083] Among them, represents the channel index of the output feature map after pointwise convolution, and represents the output feature map from the depth convolution operation, retaining the spatial information of each channel; represents the convolutional kernel weight of the pointwise convolution, and its size is , where is the number of output channels, and each convolutional kernel is responsible for fusing all input channels to generate a new output channel. represents the value of the -th output channel obtained at the position after pointwise convolution.

[0084] To better comprehensively utilize multi-modal data, the present invention adopts a multi-modal feature fusion algorithm based on deep learning to integrate and analyze the feature data obtained from epidermal electronic flexible sensors, eye trackers, and video micro-expression analysis techniques. As Figure 2 shown, the specific method is as follows:

[0085] Construct a joint cross-attention module;

[0086] To effectively achieve the collaborative interaction between multi-modal features, the present invention adopts a joint coding method based on the cross-attention mechanism to achieve cross-modal information fusion while maintaining the integrity of intra-modal features. Specifically, the joint cross-attention (JCA) module constructs the inter-modal association through a dual-path attention mechanism. First, the features of each modality are used as query vectors respectively, and the representation consistency information is obtained through cross-modal attention weight calculation; then, a linear transformation layer is used to adaptively fuse the modality features, and the complementary feature expression is strengthened through a feature weighting strategy. To further retain the underlying semantic information, a residual connection structure is introduced, and the feature retention ability of the model is enhanced through feature stacking. Finally, through the feature splicing operation, the hierarchical aggregation of the consistency features and the complementary features is realized, and a multi-modal feature space with rich semantic representations is constructed.

[0087] Suppose are the flexible sensor, eye movement, and facial micro-expression features after preprocessing and feature extraction respectively. Among them, is the number of samples, , and represent the feature dimensions of the three modalities respectively. Calculate the cross-attention matrix between every two of the three modalities, and a total of six cross-matrices need to be calculated. The cross-modal cross-attention calculation formula is as follows:

[0088] Cross-attention calculation formula for flexible sensors and eye movement modalities:

[0089] ,

[0090] ,

[0091] Cross-attention calculation formula for flexible sensors and facial micro-expression modalities:

[0092] ,

[0093] ,

[0094] Cross-attention calculation formula for eye movement and facial micro-expression modalities:

[0095] ,

[0096] ,

[0097] Among them, is the cross-attention weight matrix of the three modalities, represents the transpose operation, is the common feature dimension, is the scaling factor for stabilizing the attention score.

[0098] The cross-modal cross-attention matrix is used to quantify the correlation features between the three modalities and reflects the consistency features between the modalities.

[0099] Fuse multi-modal information;

[0100] Next, the present invention combines the information of each modality with the cross-attention matrix through linear transformation, integrates the information of the other two modalities, and further enhances the complementary information. The calculation formula is as follows:

[0101] ,

[0102] ,

[0103] ,

[0104] Among them, 、 and respectively represent the enhanced features of the three modalities after cross-modal information integration, is the cross-modal attention weight matrix for balancing the contribution values of different modalities.

[0105] To maintain the integrity of the basic features within the modality, the present invention realizes cross-modal information fusion through the residual connection between the attention map and the modality features. Specifically, the target feature vector is dynamically generated through the cross-modal attention map, and then linearly superimposed with the original modality features. This helps to align the feature representations between modalities and retain the original feature distribution using the residual structure. The formula is as follows:

[0106] ,

[0107] ,

[0108] ,

[0109] wherein, 、 and are the within-modality features obtained through attention weighting, 、 and are learnable weight matrices.

[0110] Finally, to enhance the semantic expression ability of the multi-modal features, the present invention adopts a hierarchical feature fusion strategy, enhances the spatial locality and suppresses the noise of each modality feature through a convolutional kernel (such as ), and then realizes cross-modal semantic aggregation through a feature concatenation operation. The calculation formula is:

[0111] ,

[0112] wherein, represents a convolution operation (the convolution kernel size is ) for extracting locally sensitive features and reducing the feature dimension; integrates the complementary information of each modality into a unified high-dimensional space, and finally forms a discriminative multi-modal joint representation. This design significantly improves the model's ability to capture complex emotional semantics through the dual mechanisms of feature refinement and cross-modal fusion.

[0113] Establish a classification prediction module;

[0114] After the multi-modal feature fusion is completed, assume that the feature after the three-modal fusion is , . Among them, is the output channel number of the th convolution, is the concatenation dimension of the three-way convolution operation. The fused feature is input into the fully connected layer, and a multi-layer fully connected network is designed to gradually reduce the dimension, introduce non-linearity and prevent overfitting, extract high-level semantic information through hierarchical dimensionality reduction, gradually suppress noise and enhance discriminability. The formula is as follows:

[0115] ,

[0116] ,

[0117] ,

[0118] ,

[0119] Among them, is the weight matrix of the th fully connected layer, is the bias term corresponding to the th fully connected layer, is the output of the th hidden layer after being activated by the ReLU function and regularized by Dropout.

[0120] In the design of the classification output layer of the multi-modal emotion recognition model, in order to achieve the joint recognition of emotion categories and emotion intensities, the present invention adopts a dual-branch output strategy in the classification module to separately implement the emotional analysis tasks of two dimensions: emotion category recognition and emotion intensity assessment.

[0121] The emotion category recognition module constructs an emotion category probability distribution generator based on the Softmax function. The output layer will generate a five-dimensional probability vector (pleasure, anger, disgust, fear, sadness), and the confidence value of each dimension represents the possibility intensity of the corresponding emotion category. According to the maximum a posteriori (MAP) criterion, the emotion category with the highest confidence is selected as the final recognition result. The formula is as follows:

[0122] ,

[0123] Among them, is the weight matrix of the emotion category classification layer, is the classification bias term, is a five-dimensional probability vector representing the predicted probability distribution of emotion categories (pleasure, anger, disgust, fear, sadness).

[0124] The emotion intensity assessment module uses a two-dimensional emotion space composed of valence and arousal to quantify emotion intensity. Based on the Valence-Arousal (VA) emotion model, emotion intensity is decomposed into two orthogonal dimensions. Among them, valence represents the positive and negative polarity of emotion, and arousal represents the physiological activation degree of emotion. The output layer uses a linear regression layer to predict continuous values. The formula is as follows:

[0125] ,

[0126] Among them, is the weight matrix of the emotion intensity regression layer, is the regression bias term, , representing the valence value and the arousal value . Finally, the output vector is constrained within a certain range through the Sigmoid function to ensure that it conforms to the normalization definition of the VA model.

[0127] The present invention uses emotion intensity as a key control parameter, and achieves the purpose of emotion intervention by adjusting the lighting, background music, and interactive elements in the virtual environment in real time, aiming to help patients cope with depressive emotions through emotional regulation training and psychological counseling. Specifically, the system can adjust the following parameters according to the patient's real-time emotional state:

[0128] Light intensity: According to the patient's anxiety or relaxation level, the system can dynamically adjust the lighting in the virtual scene, so as to provide a more suitable visual environment for the patient. For example, when strong anger is detected, the main color tone of the scene is switched to a cool color system, and the light intensity is reduced. The principle of color psychology is used to inhibit the excitement of the sympathetic nerve; when a sad emotion is detected, a progressive enhancement strategy is adopted to slowly increase the environmental brightness, simulating the natural dawn process to promote serotonin secretion.

[0129] Background music: The system can select or adjust the type and volume of the background music according to the patient's emotional state. The background music uses dynamic track synthesis technology. When the system detects that the patient is in a low mood, it may choose to play more soothing music to help the patient relieve stress.

[0130] Interaction mode: The system adjusts the interaction difficulty and rhythm in the virtual reality by monitoring the patient's real-time reactions. For example, for patients with difficulty in concentrating, the system will reduce the complexity of the interaction tasks to improve the patient's participation.

[0131] During the operation of the system, the emotional state is predicted by continuously collecting real-time data and repeating the multi-modal data emotion recognition. A dynamic monitoring mechanism for emotional stability is introduced in the design. Its core lies in using the sliding time window algorithm to verify the convergence of the emotional state: when the system continuously obtains high-confidence emotion classification results (confidence threshold > 85%) within a preset time window (such as 5 minutes), it switches to the corresponding VR scene according to the emotion intensity result in the emotional state and starts the preset emotion regulation plan; if a significant fluctuation in the emotional state is detected (confidence standard deviation > 15%), the window extension strategy is executed to continuously monitor until the emotional steady-state criterion is met.

[0132] Each emotional category (joy, anger, disgust, fear, sadness) corresponds to a specially designed scenario. For example, when it is detected that the user is in an angry mood, the ice and snow world scenario is switched. The main color tone uses a cold color system. A dynamic glacier crack is set in the center of the scenario (the expansion speed of the crack is positively correlated with the intensity of the emotion). The background sound includes the howling of the cold wind and the sound of ice crystal collisions, introducing temperature perception cues. The low-temperature color environment can reduce the skin conductance level, divert attention resources to disperse the angry emotion, and the cold metaphor scenario can activate the ventrolateral prefrontal cortex (vlPFC), enhancing the emotion regulation ability.

[0133] On the basis of virtual scenario regulation, the system also integrates a personalized intervention module. After the emotion regulation process ends, this module formulates a personalized emotion management and psychological counseling plan for each user by analyzing the user's emotion change data and intervention effects and combining the user's historical feedback. The intervention measures may include cognitive behavioral therapy (CBT), meditation training, deep breathing exercises, or other emotion regulation trainings. This design not only considers the immediate intervention effect of the virtual reality environment, but also helps users establish a long-term emotion management mechanism through subsequent personalized plans.

[0134] Throughout the process, the system dynamically adjusts the virtual reality scenario by continuously monitoring and analyzing the patient's real-time data, ensuring that the treatment process is highly personalized. This method not only improves the accuracy of the treatment, but also enhances the patient's immersion and treatment effect, thus achieving a more ideal diagnosis and treatment effect.

[0135] Through this technology, the present invention can provide a customized virtual reality diagnosis and treatment experience for each patient, greatly improving the effectiveness of the treatment and the satisfaction of users.

[0136] Embodiment

[0137] As Figure 1 shown, the present invention provides a method for assisting in the diagnosis and treatment of depression based on virtual reality technology, integrating multi-modal physiological and behavioral data. The method aims to provide accurate emotion monitoring, real-time feedback, and personalized treatment suggestions through the comprehensive analysis of the patient's peripheral physiological data, eye movement data, and facial micro-expressions in a virtual reality (VR) environment. The core of this method lies in the organic combination of multi-modal data and the virtual reality environment to dynamically adjust the virtual scenario to achieve better diagnosis and treatment effects for depression.

[0138] 1. The epidermal electronic flexible sensor is used for peripheral physiological data collection and feedback;

[0139] First, this method collects electrocardiogram, galvanic skin response, and body temperature data through epidermal electronic flexible sensors at the forehead and temple positions. The flexible sensors continuously obtain the patient's physiological state and combine its data with the stimuli in the virtual reality environment to provide real-time feedback. When an increase in heart rate or galvanic skin activity is detected, the system will help the patient relax by adjusting the content or atmosphere of the virtual environment (such as reducing the light intensity, changing the background music, etc.), achieving dual regulation of physiology and emotion.

[0140] 2. Eye movement tracking is used for cognitive and emotional state monitoring;

[0141] Eye movement tracking technology is used to monitor the patient's fixation points, saccades, and pupil changes in virtual reality. By analyzing this data, the distribution of the patient's attention and cognitive load are evaluated. This method combines eye movement data with the virtual reality scenario to help judge the patient's cognitive and emotional responses. When the patient fixates on a virtual object for too long or makes frequent saccades, the system can identify a potential anxiety state and make corresponding adjustments in the virtual scene, such as simplifying the scene content or adding soothing visual stimuli.

[0142] 3. High-definition cameras are used for micro-expression recognition and emotional feedback;

[0143] The high-definition camera captures the patient's facial micro-expression data to identify hidden emotional changes. This method combines micro-expression recognition with the virtual reality scenario to generate real-time emotional feedback. In the virtual scene, when the patient's anxiety or depressive mood is detected, the system will relieve these emotional states through interactive content in virtual reality (such as guiding the patient to perform breathing exercises or adjusting the scene atmosphere), thereby improving the treatment effect. The camera is installed in the facial area at the front end of the helmet, directly in front of the eyes, ensuring that the camera can capture the patient's facial micro-expression changes.

[0144] 4. Multimodal data fusion and dynamic adjustment of virtual reality scenarios;

[0145] The key to this method lies in the real-time fusion and analysis of multimodal data. By integrating peripheral physiological data, eye movement data, and micro-expression data, the patient's emotional fluctuations are accurately captured, and the treatment content is adjusted in real time in combination with virtual reality technology. Virtual reality not only provides an immersive treatment scenario for the patient but also dynamically adjusts the scene atmosphere according to physiological and behavioral data to make it more in line with the patient's psychological state. After accurate emotion prediction is obtained through analysis, it switches to the corresponding adjustment scenario for personalized diagnosis and treatment.

[0146] 5. Diagnostic feedback and personalized treatment:

[0147] When a patient uses this device for the first time, a series of baseline tests are conducted, including peripheral physiological data (heart rate, body temperature, skin conductance response), eye movement data, and micro-expression data. At the same time, in the virtual reality interaction, the patient is asked to complete some standard mental health questionnaires to obtain the patient's self-reported mental state.

[0148] Use deep learning techniques to analyze the collected data in real time and identify the physiological and behavioral markers of depression. Combine the patient's historical data and real-time feedback to identify the patient's emotional patterns and any abnormal behaviors, such as persistent low heart rate, excessive or low skin conductance activity, abnormal eye movement patterns, and micro-expressions.

[0149] Combine the patient's real-time data with the results of the psychological assessment questionnaire to generate a comprehensive diagnostic report. The report should include charts of the patient's physiological data, results of behavioral analysis, scores of psychological tests, and a preliminary diagnosis based on these data.

[0150] According to the diagnostic results and the patient's specific needs, design a personalized virtual reality treatment plan. It can include relaxation training, emotion management skills training, and cognitive behavioral therapy, etc. Cooperate with psychological counseling and necessary drug treatment to provide the patient with a comprehensive treatment experience.

[0151] During the adjuvant treatment process, monitor the patient's physiological and behavioral responses in real time, and dynamically adjust the virtual reality environment and treatment strategies according to the patient's response to the treatment content. For example, if the patient's heart rate rises or shows anxious micro-expressions in a specific virtual environment, the system can automatically adjust the environmental settings to relieve the patient's stress.

[0152] Regularly evaluate the patient's treatment progress by comparing the peripheral physiological data, eye movement data, and micro-expression data before and after treatment to evaluate the treatment effect. The patient and the doctor can access these evaluation reports to better understand the treatment effect and make adjustments to the treatment plan if necessary.

[0153] Provide long-term data tracking and support to help the patient maintain the treatment results and prevent recurrence. By continuously monitoring the patient's physiological and behavioral data, identify any potential emotional fluctuations or reappearance of depressive symptoms, and intervene in a timely manner.

[0154] In summary, the method of the present invention utilizes the integration of virtual reality technology and multi-modal physiological and behavioral data, and through the way of real-time regulation of the virtual scene, assists patients with depression in emotional management and psychotherapy, and has the advantages of personalization, dynamic adjustment, and immersive feedback.

[0155] In a second aspect, the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the foregoing personalized virtual reality assisted diagnosis and treatment method based on multi-modal data fusion.

[0156] In a third aspect, the present invention provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor can be caused to implement the foregoing personalized virtual reality assisted diagnosis and treatment method based on multi-modal data fusion.

[0157] The specific embodiments described above further elaborate on the purpose, technical solution and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A virtual reality-assisted diagnosis and treatment method for depression that integrates multimodal data, characterized in that: The method comprises: Step 1: Obtain peripheral physiological data through epidermal electronic flexible sensors, obtain eye movement data through eye trackers, and obtain micro-expression data through high-precision cameras; Step 2: preprocessing and feature extraction of the peripheral physiological data, eye movement data and micro-expression data to obtain multimodal feature data; Step 3: Use a multimodal deep learning model based on an attention mechanism to perform fusion analysis on the multimodal feature data to identify the patient's emotional state; Step 4: Dynamically adjust the virtual reality scene based on the identified emotional state to provide personalized emotion regulation and auxiliary treatment.

2. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: In step 1, the peripheral physiological data include the patient's heart rate, body temperature and skin electrical activity data, the eye movement data include the patient's gaze point position, gaze duration, eye movement trajectory, blinking frequency and pupil diameter change data, and the micro-expression data include the patient's facial micro-expression change data.

3. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: In step 2, for the peripheral physiological data, a Butterworth filter is used to remove high-frequency noise, and a bidirectional long short-term memory network is used to process the filtered peripheral physiological data to extract peripheral physiological time features.

4. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: In step 2, the eye movement data is corrected and smoothed using a Savitzy-Golay filter, the filtered eye movement data is reduced in dimension to obtain low-dimensional features using a principal component analysis method, and a feature extraction network based on a convolutional neural network is used to perform deep feature extraction on the low-dimensional features to obtain a comprehensive eye movement feature vector.

5. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: In step 2, the ASM algorithm is used to locate the key feature points of the face for the micro-expression data, and the facial key feature point data extracted by deep convolution and point-by-point convolution of the DSCNNs network are used to obtain the facial fusion features.

6. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: In step 3, using a multimodal deep learning model based on an attention mechanism to perform fusion analysis on the multimodal feature data includes: The peripheral physiological time feature, the eye movement comprehensive feature vector, and the facial fusion feature are used as query vectors respectively, and the cross-attention matrices corresponding to the peripheral physiological time feature, the eye movement comprehensive feature vector, and the facial fusion feature are obtained through cross-modal attention weight calculation; Based on the respective corresponding cross attention matrices, a linear transformation layer is used to adaptively fuse the peripheral physiological time features, the eye movement comprehensive feature vector, and the facial fusion features; The residual connection structure is introduced to superimpose the adaptive fusion features with the peripheral physiological time features, eye movement comprehensive feature vectors, and facial fusion features. The superimposed features are hierarchically aggregated through feature concatenation operations to construct a multimodal feature space with rich semantic representation.

7. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: Identifying the patient's emotional state in step 3 includes identifying emotion categories and emotion intensity. The emotion category identification constructs an emotion category probability distribution generator based on the Softmax function, and the output layer generates a five-dimensional probability vector including pleasure, anger, disgust, fear, and sadness. The confidence value of each dimension represents the possibility intensity of the corresponding emotion category; the emotion intensity assessment uses a two-dimensional orthogonal emotion space composed of valence and arousal value to quantify the emotion intensity, wherein the valence represents the positive and negative polarity of the emotion, and the arousal value represents the physiological activation degree of the emotion.

8. The method for virtual reality-assisted diagnosis and treatment of depression by integrating multimodal data according to claim 1, characterized in that: The step 4 includes using a sliding time window algorithm to verify the convergence of the emotional state. When the system continuously obtains high-confidence emotion classification results within a preset time window, it switches to the corresponding VR scene according to the emotion intensity result in the emotional state and starts the preset emotion regulation scheme. If significant fluctuations in the emotional state are detected, the window extension strategy is executed and continuous monitoring is implemented until the emotional homeostasis criteria are met.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; Among them, when one or more programs are executed by the one or more processors, the one or more processors implement the virtual reality assisted diagnosis and treatment method for depression that integrates multimodal data as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by the processor, the processor can implement the virtual reality assisted diagnosis and treatment method for depression that integrates multimodal data as described in any one of claims 1-8.

Images