A social simulation digital intervention system based on virtual reality

Abstract

The present invention belongs to the field of neuroscience and medical rehabilitation, and proposes a social simulation digital intervention system based on virtual reality. The system includes: a data acquisition module for collecting physiological data, psychological and behavioral data; a data preprocessing module for cleaning and standardizing the collected raw data; a data processing and analysis module for processing and analyzing the user's physiological data, psychological and behavioral data through multimodal data fusion technology; an electrical stimulation intervention module for interfering with electrical stimulation of the user during social simulation training to adjust the user's neural activity state; a dynamic training adjustment module for dynamically adjusting the difficulty and focus of the training content according to the stimulation effect, the user's behavioral data, etc.; an optimization module for adjusting the weight vector by optimizing the loss function to minimize the error. This solution comprehensively evaluates the patient's social cognition and emotional changes, and provides key information support for personalized intervention.

Description

Technical Field

[0001] The present invention belongs to the field of neuroscience and medical rehabilitation, and in particular relates to a social simulation digital intervention system based on virtual reality. Background Art

[0002] In real life, some patients with social disorders or social anxiety disorders often find it difficult to interact with others because they cannot correctly understand social situations and the intentions of others and lack effective social skills, which seriously affects their quality of life. For such patients, targeted social skills training is needed to enhance their social cognitive ability, emotional management ability and social interaction skills. With the development of virtual reality (VR) technology, VR-based social simulation training has gradually been applied to such fields. By constructing realistic virtual social scenes, different social situations can be simulated and repeatedly experienced, and patients can conduct social training in a safe and controllable virtual environment to improve their social skills. However, the existing VR social simulation training system has some shortcomings. Due to the lack of real-time monitoring of the patient's physiological and psychological state, it is difficult to fully evaluate the patient's cognitive and emotional changes, and it is impossible to dynamically adjust the training content and difficulty according to individual differences, so the training effect is limited. In addition, the existing system mainly relies on audio-visual simulation, lacks multimodal fusion of perceptual channels, and lacks immersive experience. Summary of the invention

[0003] In view of this, the present invention provides a social simulation digital intervention system based on virtual reality, the system comprising:

[0004] Data collection module, used to collect physiological data, psychological and behavioral data as raw data;

[0005] A data preprocessing module is used to clean and standardize the collected raw data to eliminate noise and invalid data;

[0006] Data processing and analysis module, used to process and analyze the user's physiological data, psychological and behavioral data through multimodal data fusion technology to evaluate the user's social behavior and emotional state and provide feedback;

[0007] An electrical stimulation intervention module, used to perform non-invasive transcutaneous vagus nerve intervention electrical stimulation on the user during social simulation training to regulate the user's neural activity state;

[0008] The dynamic training adjustment module dynamically adjusts the difficulty and emphasis of the training content according to the stimulation effect of the electrical stimulation intervention module, the difference between the user's behavior data and the ideal data, and the training scenario;

[0009] The optimization module is used to adjust the weight vector to minimize the error by optimizing the loss function.

[0010] In particular, the data acquisition module includes an eye tracking device, a facial expression analyzer, a hand motion capture device, a somatosensory detection device, a speech recognition system and a neural activity monitoring device.

[0011] In particular, the multimodal data fusion process calculates the user's social behavior and emotional state response through the following formula:

[0012]

[0013] in, For in time t The social behavior and emotional state response values ​​of the place;

[0014] n is the modality number of physiological data, For the i Physiological data in time The response function at For the i The weight factors corresponding to the physiological data;

[0015] m is the number of modalities of psychological and behavioral data, For the j Psychological and behavioral data in time The response function at For the j Weight factors corresponding to various psychological and behavioral data.

[0016] In particular, the physiological data includes: eye movement data, facial expression data, somatosensory data and neural activity data;

[0017] The psychological and behavioral data include: hand movements, gesture data and voice data.

[0018] In particular, the dynamic training adjustment module is dynamically adjusted by the following formula:

[0019]

[0020] in, is the adjustment result of the current training scene, is the vector of the user's physiological data, psychological and behavioral data in the current m+1th scenario; For the first time in history The ideal physiological data, psychological and behavioral data indicators set in each scenario, is the kernel function, which measures the similarity between the current scene and the historical scene; Represents the feature vector of the current training scene, where: represents the number of previous training scenes, Indicates the total number of historical and current scenarios considered; Indicates the history scene feature vector; For users in time t The social behavior and emotional state response values ​​at is the weight factor of the training scenario, is the response function of the electrical stimulation intervention effect; is the weight factor of the electrical stimulation intervention effect; is the offset, used to adjust the training difficulty; is the weight factor of the user's physiological data, psychological and behavioral data, n is the number of historical scenes available for reference.

[0021] In particular, the optimization module adjusts the weight vector by optimizing the loss function as follows:

[0022]

[0023] in is the loss function, ω is the weight vector, is the error vector of the i-th data point, γ is the regularization parameter, and N is the number of data points; by optimizing the loss function , adjust the weight ω so that the sum of the square of the error and the weight is minimized; according to the optimized weight ω and error e, dynamically adjust the training content and difficulty.

[0024] In particular, the system performs a variety of social skills training in a virtual reality environment, including social initiation skills training, social response skills training, emotion expression and management training, and social conflict handling training.

[0025] In particular, the electrical stimulation intervention module includes: stimulation electrodes for applying weak current to the concha area of ​​the user;

[0026] Current generator, used to generate weak AC or pulse current signal;

[0027] A stimulation parameter adjustment module is used to adjust the stimulation parameters according to the user's social behavior and emotional state response value, the difference between the user's physiological data, psychological and behavioral data in the current scene and the ideal physiological data, psychological and behavioral data set in various scenes in history, the difficulty level of the current training content and the user's electrical stimulation response history record, and the similarity between the current scene and the historical scene;

[0028] The synchronization control module is used to synchronize non-invasive transcutaneous vagus nerve intervention electrical stimulation with the virtual reality social scene and apply stimulation in a specific social situation.

[0029] Beneficial effects:

[0030] 1. Through the technical solution of the present invention, a social simulation digital intervention system based on virtual reality is realized, which can construct realistic virtual social scenes, provide a safe and controllable social training environment for patients with social disorders, and improve the realism and immersion of training.

[0031] 2. Through the technical solution of the present invention, physiological sensors, computer vision, speech recognition and other technologies are combined with virtual reality to achieve multimodal perception and fusion of patients' behavior, physiology and neural activity status, comprehensively evaluate patients' social cognition and emotional changes, and provide key information support for personalized intervention.

[0032] 3. Through the technical solution of the present invention, a non-invasive transcutaneous vagus nerve stimulation (tVNS) intervention technology is introduced, which can adjust the non-invasive transcutaneous vagus nerve stimulation stimulation parameters in real time according to the patient's neural activity response, regulate the patient's brain activity, optimize their cognitive state and emotional management ability, and improve the pertinence and efficiency of social training.

[0033] 4. Through the technical solution of the present invention, a behavioral indicator analysis model and a dynamic training adjustment algorithm are established, which can dynamically adjust the difficulty and focus of the virtual reality training content according to the difference between the patient's behavioral performance and the ideal indicators, the neural activity response and the tVNS stimulation effect, forming a closed-loop adaptive training system.

[0034] 5. Through the technical solution of the present invention, multidisciplinary knowledge such as virtual reality technology, multimodal interaction technology, neuroscience and artificial intelligence algorithms are comprehensively utilized, and advanced virtual simulation, perception fusion and neural regulation technologies are applied to the field of social cognitive training, expanding the application scope of related technologies.

[0035] 6. Through the technical solution of the present invention, a new digital intervention method is provided for patients with social disorders, which overcomes the limitations of traditional training methods and can provide customized and efficient social skills training according to the individual differences of patients, thereby effectively improving the quality of life of patients, and has good social benefits and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention provides a social simulation digital intervention system based on virtual reality. DETAILED DESCRIPTION

[0037] The present invention is described in detail below with reference to the accompanying drawings and examples. The present invention provides a social simulation digital intervention system based on virtual reality, such as Figure 1As shown, the system includes:

[0038] The data acquisition module is used to collect physiological data, psychological and behavioral data; the data acquisition module includes an eye tracking device, a facial expression analyzer, a hand motion capture device, a body sensing detection device, a speech recognition system and a neural activity monitoring device.

[0039] The data preprocessing module is used to clean and standardize the collected raw data and eliminate noise and invalid data. The data preprocessing module performs the following processing on the collected raw data:

[0040] Data cleaning is used to remove obvious abnormal values ​​and outliers, fill in missing or lost data, and remove duplicate data; standardization is used to non-dimensionalize data of different dimensions and map the data to the same numerical range; One-Hot encoding or ordered encoding is performed on classified data. Noise elimination is used to filter, smooth, and perform other noise reduction processing on image and video data such as eye movements and facial expressions; and spectrum analysis, wavelet denoising, and other processing are performed on time series data such as speech and neural activity.

[0041] Then, the preprocessed data is divided into training set, validation set and test set through data segmentation;

[0042] The data processing and analysis module is used to process and analyze the user's physiological data, psychological and behavioral data through multimodal data fusion technology to evaluate the user's social behavior and emotional state and provide feedback.

[0043] The data processing and analysis module processes and analyzes the user's physiological data, psychological and behavioral data through multimodal data fusion technology to evaluate the user's social behavior and emotional state, wherein the multimodal data fusion technology calculates the user's social behavior and emotional state response through the following formula:

[0044]

[0045] in, For in time t The social behavior and emotional state response values ​​of the place;

[0046] n is the modality number of physiological data, For the i Physiological data in time The response function at For the i Weight factors corresponding to the physiological data;

[0047] m is the number of modalities of psychological and behavioral data, For the jPsychological and behavioral data in time The response function at For the j Weight factors corresponding to various psychological and behavioral data.

[0048] The physiological data include: eye movement data, facial expression data, body sensation data and neural activity data; the neural activity data include: electroencephalogram (EEG) and / or near-infrared fNIRS data.

[0049] The psychological and behavioral data include: hand movements, gesture data and voice data.

[0050] An electrical stimulation intervention module, used to perform non-invasive transcutaneous vagus nerve stimulation (tVNS) intervention on the user during social simulation training to regulate the user's neural activity state;

[0051] The dynamic training adjustment module dynamically adjusts the difficulty and emphasis of the training content according to the stimulation effect of the electrical stimulation intervention module, the difference between the user's behavior data and the ideal data, and the training scenario; the dynamic training adjustment module performs dynamic adjustment through the following formula:

[0052]

[0053] in, is the adjustment result of the current training scene, is the vector of the user's physiological data, psychological and behavioral data in the current m+1th scenario; For the first time in history The ideal physiological data, psychological and behavioral data indicators set in each scenario, is the kernel function, which measures the similarity between the current scene and the historical scene; Represents the feature vector of the current training scene, where: represents the number of previous training scenes, Indicates the total number of historical and current scenarios considered; Indicates the history scene feature vector; For users in time t The social behavior and emotional state response values ​​at is the weight factor of the training scenario, is the response function of the electrical stimulation intervention effect; is the weight factor of the electrical stimulation intervention effect; is the offset, used to adjust the training difficulty; is the weight factor of the user's physiological data, psychological and behavioral data, n is the number of historical scenes available for reference.

[0054] The electrical stimulation intervention module includes: stimulation electrodes for applying weak current to the user's concha area;

[0055] Current generator, used to generate weak AC or pulse current signal;

[0056] A stimulation parameter adjustment module is used to adjust the stimulation parameters according to the user's social behavior and emotional state response value, the difference between the user's physiological data, psychological and behavioral data in the current scene and the ideal physiological data, psychological and behavioral data set in various scenes in history, the difficulty level of the current training content and the user's electrical stimulation response history record, and the similarity between the current scene and the historical scene;

[0057] The synchronization control module is used to synchronize non-invasive transcutaneous vagus nerve intervention electrical stimulation with the virtual reality social scene and apply stimulation in a specific social situation.

[0058] When the user's social behavior and emotional state response value R(t) When levels are below normal, increase the intensity and frequency of stimulation to increase the user's concentration and arousal;

[0059] When the training content is more difficult, the stimulation intensity is appropriately increased to improve the user's concentration and coping ability; according to the user's electrical stimulation response history, the stimulation parameters are adjusted to achieve the personalized best stimulation effect.

[0060] The optimization module is used to adjust the weight vector by optimizing the loss function to minimize the error. The optimization module adjusts the weight vector by optimizing the loss function as follows:

[0061]

[0062] in is the loss function, ω is the weight vector, is the error vector of the i-th data point, γ is the regularization parameter, and N is the number of data points; by optimizing the loss function , adjust the weight ω so that the sum of the square of the error and the weight is minimized; according to the optimized weight ω and error e, dynamically adjust the training content and difficulty.

[0063] The system can perform various social skills training in a virtual reality environment, including social initiation skills training, social response skills training, emotion expression and management training, and social conflict handling training.

[0064] The social initiation skills training includes how to proactively greet others and start a conversation; the social response skills training includes how to appropriately respond to others' greetings or inquiries; the emotional expression and management training includes how to identify and express one's own emotions, and how to manage negative emotions.

[0065] The present invention proposes a social simulation digital intervention system based on virtual reality, which organically combines multidisciplinary knowledge such as virtual reality technology, multimodal interaction technology, neural regulation technology and artificial intelligence algorithm to form a complete digital intervention solution for efficient and personalized social skills training for patients with social disorders.

[0066] The system is centered on virtual reality and can build realistic and immersive virtual social scenes, providing patients with a safe and controllable social training environment. It simulates different social situations and supports repeated practice, greatly improving the realism and experience of training.

[0067] At the same time, the system integrates technologies such as physiological sensors, computer vision and speech recognition, realizing multimodal perception and fusion of patients' behavior, physiology and neural activity status, comprehensively evaluating patients' cognitive, emotional and behavioral performance in the social process, and providing data support for personalized intervention.

[0068] In addition, the system also introduces non-invasive transcutaneous vagus nerve stimulation (tVNS) intervention technology, which regulates the patient's brain activity by adjusting tVNS stimulation parameters in real time, optimizes their attention, emotion management, social cognition and other abilities, and improves the targetedness and efficiency of social training.

[0069] The system has established a behavioral indicator analysis model and a dynamic training adjustment algorithm, which can dynamically adjust the difficulty and focus of virtual reality training content according to the patient's behavioral performance, neural activity response, and tVNS stimulation effect, forming a closed-loop adaptive training system to continuously optimize the intervention plan.

[0070] By integrating and applying multiple advanced technologies such as virtual reality simulation, multimodal interaction, neural regulation and artificial intelligence algorithms, the system of the present invention achieves highly personalized, intelligent and efficient social training, providing a new digital intervention method for patients with social disorders, overcoming the limitations of traditional training methods, and being able to provide customized social skills training based on individual differences of patients, thereby effectively improving the quality of life of patients. It has broad application prospects.

[0071] This design achieves dynamic training adjustment through optimization methods, combines the user's neural activity response with behavioral data, adjusts the training content and difficulty in real time, and provides personalized, real-time feedback and optimization for high-functioning autistic patients. The training strategy is further precisely adjusted by optimizing the loss function to improve the training effect. Through this dynamic training adjustment design, the system can generate personalized feedback based on the user's real-time performance and neural activity response in the virtual reality environment, and dynamically adjust the training content and difficulty using optimization methods. This can help high-functioning autistic patients gradually improve their social skills and provide therapists with effective monitoring and guidance tools.

[0072] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0073] It is obvious to those skilled in the art that the embodiments of the present invention are not limited to the details of the above exemplary embodiments, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present invention is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the embodiments of the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules or devices stated in the system, device or terminal claims can also be implemented by the same unit, module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0074] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the embodiments of the present invention and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred implementation modes, those skilled in the art should understand that the technical solutions of the embodiments of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A social simulation digital intervention system based on virtual reality, characterized in that: The system includes: Data collection module, used to collect physiological data, psychological and behavioral data as raw data; A data preprocessing module is used to clean and standardize the collected raw data to eliminate noise and invalid data; Data processing and analysis module, used to process and analyze the user's physiological data, psychological and behavioral data through multimodal data fusion technology to evaluate the user's social behavior and emotional state; An electrical stimulation intervention module, used to perform non-invasive transcutaneous vagus nerve intervention electrical stimulation on the user during social simulation training to regulate the user's neural activity state; A dynamic training adjustment module, used to dynamically adjust the difficulty and emphasis of the training content according to the stimulation effect of the electrical stimulation intervention module, the difference between the user's physiological data, psychological and behavioral data and the set ideal data, and the training scenario; The optimization module is used to adjust the weight vector to minimize the error by optimizing the loss function; The data acquisition module includes an eye tracking device, a facial expression analyzer, a hand motion capture device, a body sensing detection device, a speech recognition system, and a neural activity monitoring device; The multimodal data fusion technology calculates the user's social behavior and emotional state response through the following formula: Where R(t) is the response value of social behavior and emotional state at time t; n is the number of modalities of physiological data, f i (t) is the response function of the ith physiological data at time t, a i is the weight factor corresponding to the i-th physiological data; m is the number of modes of psychological and behavioral data, g is j (t) is the response function of the jth psychological and behavioral data at time t, b j is the weight factor corresponding to the jth psychological and behavioral data; Wherein, the physiological data includes: eye movement data, facial expression data, body sensation data and neural activity data; The psychological and behavioral data include: hand movements, gesture data and voice data; the neural activity data include: electroencephalogram (EEG) and / or near infrared fNIRS data; The dynamic training adjustment module is dynamically adjusted by the following formula: Among them, X m+1 is the adjustment result of the current training scene, u m+1 is the vector of the user's physiological data, psychological and behavioral data in the current m+1th scenario; is the ideal physiological data, psychological data and behavioral data index set in the k-th scenario in history, K(x m+1 , x k ) is the kernel function, which measures the similarity between the current scene and the historical scene; x m+1 represents the feature vector of the current training scenario, where: m represents the number of previous training scenarios, and m+1 represents the total number of historical and current scenarios considered; x k represents the kth historical scene feature vector; R(t) is the social behavior and emotional state response value of the user at time t, β1 is the weight factor of the training scene, f(S) is the response function of the electrical stimulation intervention effect; β2 is the weight factor of the electrical stimulation intervention effect; v is the offset, which is used to adjust the training difficulty; β3 is the weight factor of the user's physiological data, psychological and behavioral data, and n is the number of historical scenes that can be referenced; The optimization module adjusts the weight vector by optimizing the loss function as follows: Where J(ω,e) is the loss function, ω is the weight vector, and e i is the error vector of the ith data point, γ is the regularization parameter, and N is the number of data points. By optimizing the loss function J(ω,e), the weight ω is adjusted to minimize the sum of the squares of the error and the weight. According to the optimized weight ω and error e, the difficulty and emphasis of the training content are dynamically adjusted.

2. The social simulation digital intervention system based on virtual reality according to claim 1, characterized in that: in, The system performs various social skills training in a virtual reality environment, including social initiation skills training, social response skills training, emotion expression and management training, and social conflict handling training.

3. The social simulation digital intervention system based on virtual reality according to claim 2, characterized in that: in, The electrical stimulation intervention module includes: stimulation electrodes for applying weak current to the user's concha area; Current generator, used to generate weak AC or pulse current signal; A stimulation parameter adjustment module is used to adjust the stimulation parameters according to the user's social behavior and emotional state response value, the difference between the user's physiological data, psychological and behavioral data in the current scene and the ideal physiological data, psychological and behavioral data set in various scenes in history, the difficulty level of the current training content and the user's electrical stimulation response history record, and the similarity between the current scene and the historical scene; The synchronization control module is used to synchronize non-invasive transcutaneous vagus nerve intervention electrical stimulation with the virtual reality social scene and apply stimulation in a specific social situation.

Images