Virtual Reality Therapy System
The VR treatment system addresses the lack of resources for mental disorder treatment by using a head-mounted display and machine learning to adapt treatment scenarios and provide personalized feedback, improving therapeutic efficacy.
Patent Information
- Application Number
- JP2021575962
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-06-17
- Filing Date
- 2020-06-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-06-05
AI Technical Summary
There is a lack of adequate resources for treating mental disorders using virtual reality (VR) systems, which can provide immersive and interactive environments for psychotherapy.
A VR treatment system is developed, incorporating a head-mounted display unit, input devices, and computers to present treatment scenarios, measure mental anxiety states, and provide feedback through a coach character, utilizing machine learning to adapt the treatment based on patient interactions and biometric data.
The system effectively provides immersive psychotherapy by adapting to patient responses, offering personalized treatment scenarios and feedback, enhancing therapeutic outcomes for mental disorders.
Smart Images

Figure 0007581257000001 
Figure 0007581257000002 
Figure 0007581257000003
Abstract
Description
[Technical field]
[0001] Field The present specification relates to a virtual reality (VR) therapy system for providing psychotherapy to a patient. [Background technology]
[0002] background As used herein, a VR system includes a virtual reality headset, i.e., a head-mounted display unit that provides a simulated, immersive virtual environment to a user.
[0003] Background prior art regarding the use of such techniques can be found in US6,425,764; US2018 / 0190376; WO2019 / 036051, and Lancet Psychiatry 2018, 5: 625-32 Freeman et al. Summary of the Invention [Means for solving the problem]
[0004] overview This document describes the use of a VR system for the treatment of psychiatric disorders, which are common disorders but for which resources for treatment are often scarce.
[0005] Thus, in a first aspect, a virtual reality (VR) therapeutic system for providing psychotherapy to a patient is provided. In an implementation, the VR therapeutic system comprises a VR system for providing a virtual environment and one or more computers coupled to the VR system or a part thereof. The VR system may include at least, for example, a head-mounted display unit having a voice generating and / or detecting capability, and one or more VR input devices, for example, a hand controller system for tracking the patient's hand movements, such as a 3D mouse, a glove, a handheld control device, or a hand tracking sensor. The VR system, for example, the head-mounted display, may include a head motion tracking system. The one or more computers coupled to the VR system may be workstations, or in a standalone VR system, they may be integrated into the head-mounted display unit (headset), or they may be remote from the headset, for example, in the cloud.
[0006] In an implementation, the one or more computers may be configured to present a set of treatment scenarios to the patient in the virtual environment using the VR system, for example, to simulate a real-world scenario in which the patient finds difficulty. Each treatment scenario may be defined by a spatial region in the virtual environment. Each treatment scenario may include at least one scenario task to be performed by the patient in the virtual environment by interacting with the VR system, which may be referred to as a patient activity task because it involves performing an activity, e.g., speaking, eating, or physical activity, by the patient in the virtual environment. Thus, at least one of the scenario tasks may include at least a dialogue task in which the patient character interacts with a task character in the virtual environment, or a non-verbal, e.g., physical activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment.
[0007] The patient may be represented in the virtual environment by a patient character. Typically, only a portion of the patient is represented, e.g., the patient's hands and / or arms. During at least a portion of the time, e.g., when prompts are being delivered to the patient, the virtual environment may include a representation of a coach character. This representation may be a 3D representation of the coach character in the virtual environment, or may be a representation of an aspect of the coach character, such as the coach character's hands or footprints.
[0008] The one or more computers may be further configured to determine a psychological anxiety state parameter representative of the patient's psychological anxiety state. This may be performed by measuring one or more characteristics of an interaction between the patient character and a task character or one or more objects in the virtual environment. The one or more computers may be further configured to monitor performance of the scenario task in the virtual environment.
[0009] In response to non-performance of the scenario task after, for example, a certain amount of time and / or the patient's mental anxiety state as indicated by the mental anxiety state parameter, the system may assist the patient in completing the scenario task in the virtual environment by providing VR feedback using a coach character. The VR feedback may include verbal prompts, i.e., spoken by the coach character, and / or visual prompts, provided in the virtual environment in response to, for example, detecting the patient's mental anxiety state. The prompts may include instructions or guidance for performing the task and / or may include spoken words, for example to calm or encourage the patient.
[0010] In some implementations, the one or more computers may determine the psychological anxiety state parameter by performing one or more measurements in the virtual environment. The measurements in the virtual environment may include determining a behavior or characteristic of a patient character in the virtual environment. In some implementations, the measurements in the virtual environment do not include performing a biometric measurement on the patient in the real world.
[0011] For example, the one or more computers may measure the proximity of the patient character to the task character, e.g., by reading the respective positions of the task character and the patient character and determining a distance metric between them, or may measure the proximity to task-related objects in the virtual environment. Additionally or alternatively, the VR system may include an eye-tracking module, and the one or more computers may measure, i.e., determine, gaze direction in the virtual environment. For example, in the case of interaction with a task character, this may be used to determine when the patient (character) is looking down, e.g., at the feet of the task character, or looking away from the task character. Similarly, gaze direction may be used to determine when gaze is averted from a difficult or unpleasant task.
[0012] In another example, the one or more computers may measure the time it takes for a patient character (i.e., the patient) to respond to a verbal expression of a task character, or the time it takes for the patient character (i.e., the patient) to perform an activity with respect to one or more objects. This may establish when the patient (character) is resistant to performing a task or is slow to respond. Some or all of these measurements may be used to determine or adjust the value of a mental anxiety state parameter, for example, increasing (or decreasing) the value of the parameter in response to the measurements. Additionally or alternatively, the one or more computers may measure the body posture of the patient character in the virtual environment.
[0013] Additionally or alternatively, the system may monitor performance of a scenario task, for example, by measuring the amount of time that has elapsed since starting a task, or by determining whether the patient (character) speaks in response to a prompt, for example a voice prompt from the task character. In an implementation, the system may detect whether the patient is speaking, but need not perform speech recognition.
[0014] In some implementations, the VR therapeutic system may further perform one or more real-world biometric measurements and use these to determine the mental anxiety state parameter. For example, one or more of the measurements, whether performed in the real world or in a virtual environment, may be combined to determine the mental anxiety state parameter, for example, using a weighted combination of the measurements (optionally scaled or normalized). The weight of the combination may be determined by routine experimentation or automatically, for example, using machine learning. The real-world biometric anxiety measurement may be determined using a handheld controller of the VR therapeutic system for connection to the patient. Such real-world biometric anxiety measurements may include, for example, measurements of galvanic skin response, heart rate, cortisol levels, or movement / activity levels, as described further below. Another example of an anxiety measurement includes measurements of the patient's pupil size / dilation, which may be measured using eye-tracking or similar devices. Further examples of anxiety measurements include measurements of the patient's tremors, rapid hand movements, or more general body movements, for example, performed using a handheld controller. Such measurements may also be used when anxiety measurements are described later.
[0015] In some implementations, a patient state neural network (or other machine learning subsystem) is provided to determine the mental anxiety state parameter. Input to the patient state neural network may include one or more measurements in the virtual environment and / or one or more real-world biometric measurements. The patient state neural network may output the mental anxiety state parameter directly, or the patient state neural network may output a set of scores or probability values, one for each of a set of classification categories that may represent the degree of mental anxiety of the patient. These scores may be used to classify the patient as anxious or not anxious and / or grade the patient's anxiety into three or more categories. The patient state neural network may be trained using supervised learning techniques, for example, based on the patient's reported anxiety level, or based on a more objective measure established using existing techniques. For example, anonymized data from previous patients with similar associated scores. Alternatively, the patient state neural network may be trained using supervised learning techniques. In an implementation, the patient state neural network (classification) output determines a psychological anxiety state parameter that is used to control the coach character, so that it forms part of a closed loop feedback system (to the patient), which may be used to adjust or control the patient's experience in the VR environment, for example, to calm the patient with speech output when patient anxiety is detected. The neural network output may also be used, for example, to predict the patient's response to a treatment or individual scenario, and / or to select a scenario or set of scenarios with a high (highest) probability of a positive outcome.
[0016] In some implementations, the psychological anxiety state parameter may be used to determine the patient's psychological anxiety state by comparing the value of the psychological anxiety state parameter to a first threshold, which may be fixed, e.g., a predetermined value, or may be variable depending on the patient and / or scenario and / or history (e.g., a moving average). Additionally or alternatively, a change in the value of the psychological anxiety state parameter may be detected, e.g., above a second threshold, which may also be fixed or variable as described above. Additionally or alternatively, the value of the psychological anxiety state parameter may be processed using an explicit or implicit pattern matching algorithm to determine the patient's psychological anxiety state. For example, the value of the psychological anxiety state parameter may be provided to a trained neural network classifier, e.g., trained to provide anxious and non-anxious classification outputs. Such a classifier may be trained by supervised learning using patients with other anxiety measurements, e.g., heart rate and / or galvanic skin response (GSR) and / or cortisol levels and / or movement or activity levels. In other implementations, the classifier is trained using unsupervised learning. Alternatively, another machine learning based classifier such as a support vector machine (SVM) or random forest may be employed.
[0017] In an implementation, the feedback from the coach character includes speech provided to the patient via a headset voice generation system. The VR system may include a patient mobile device configured to monitor one or both of the patient activity and the degree of patient anxiety when the patient is in the real world and away from the virtual environment. For example, the patient mobile device may comprise or consist of a wearable device such as a wristband configured to perform anxiety measurements, e.g., heart rate and / or galvanic skin response (GSR) and / or cortisol level and / or patient movement or activity level (representing the degree of physical movement of the patient, which may, but need not, include the patient's movement from one location to another), using corresponding sensors. The patient mobile device may process the anxiety measurements, e.g., heart rate / GSR / cortisol level / movement or activity level data, to detect the patient's anxiety, e.g., as described above (by comparison to a threshold or threshold change, or by pattern matching). Cortisol levels may be detected in sweat from the patient using a wearable sensor, for example, as described in "Molecularly selective nanoporous membrane-based wearable organic electrochemical device for noninvasive cortisol sensing", Parlak et al., Science Advances 20 Jul 2018 Vol 4(7), DOI 10.1 126 / sciadv.aar2904. Additionally or alternatively, the patient mobile device may detect the location of the device and use the location to identify when the patient is in a real-world scenario that corresponds to one of the treatment scenarios in the virtual environment. The patient mobile device may include, for example, a mobile phone that is optionally coupled to the smart wristband.
[0018] Thus, the Patient Mobile Device may utilize the monitored patient activity / anxiety to identify when the patient is in a stressful real-world situation (which may, but need not, correspond to one of the treatment scenarios). In response, the Patient Mobile Device may provide real-world feedback, i.e., feedback when the patient is in the real world and away from the virtual environment. The real-world feedback may include spoken audio output to the patient, e.g., via headphones, earphones, etc., in the voice used by the coach character in the virtual environment. For example, the spoken audio output may include any words spoken using the voice of the coach character and may include soothing words and / or spoken instructions or encouragement.
[0019] The patient mobile device may communicate with one or more computers, for example via a remote server that may store / process patient-related data, for example in encrypted form, in a cloud. Optionally, the one or more computers may be configured to capture patient data from the patient while the patient is in a treatment scenario in the virtual environment and provide this data, for example in processed or encrypted form, to the patient mobile device, for example via a remote server. The patient mobile device may be configured to provide real-world feedback using the captured data. For example, the captured data may include data related to the patient's anxiety level, for example data derived from a mental anxiety state parameter, which may then be used to determine when the patient has an elevated anxiety level in the real world. Thus, a real-world anxiety measure may be derived, for example by machine learning, from data of a particular patient or cohort of patients. Additionally, a virtual world coach character with which the patient is familiar may be transported to the real world to provide coaching guidance, typically by playing the voice of the coach character, but potentially by playing a representation of the coach character, in challenging real-world scenarios.
[0020] The set of treatment scenarios may include treatment scenarios of the same type, e.g., paying for items, meeting people, etc., but may include different difficulty levels (see below) and / or they may include different types of treatment scenarios. In general, however, each treatment scenario is configured to simulate a real-world scenario that the patient finds difficult, each having a set of increasing difficulty levels. In an implementation, one or more computers are configured to provide a welcome room VR scenario (not one of the treatment scenarios) that includes a coach character. In the welcome room, the patient character may be enabled to select one of the treatment scenarios and / or a difficulty level of the selected treatment scenario, facilitated, for example, by the coach character. In some implementations, the difficulty level is equal to or greater than a previously completed difficulty level for a treatment scenario (of the same type), and in other implementations, the patient may be allowed to retry the scenario of the same or lower difficulty. Potential progression through the difficulty levels may also be based on or controlled by, for example, the patient's real-world responses in similar scenarios.
[0021] In some implementations, treatment scenarios of different difficulty levels may be procedurally constructed, which may reduce the memory requirements of the system. For example, the difficulty level of a selected treatment scenario may be changed by modifying one or more of the background noise level, the number of (task) characters in the selected treatment scenario, the duration of the treatment scenario / task, the location of the task in the virtual environment, and the behavior of the task characters.
[0022] For example, to increase the difficulty level, the task character may be configured to make direct eye contact with the patient character, and conversely, to decrease the difficulty level, the task character may be configured to not make direct eye contact with the patient character. Alternatively, the frequency of eye contact between the task character and the patient character may be altered. Eye contact may be determined according to whether the task character's gaze is directed at the patient character's eyes (i.e., whether the patient character is being observed), which may require, but need not require, that the patient character look at the task character. Thus, in some implementations, the behavior of the task characters is altered by altering the number of task characters that observe the patient character, e.g., by altering a parameter associated with each task character that defines whether the task character is configured to observe the patient character. In another example, the behavior of the task characters is altered by the body posture of the task characters. In a further example, the location of the task in the virtual environment may be altered, e.g., by altering a parameter that defines the location of the task. A task in a central area of the virtual environment, away from walls or other boundaries, may be more difficult than a task performed adjacent to a wall or other boundary (which is less likely to be observed). Such a technique may use less memory than storing each task / configuration separately, as well as ease task definition and reduce the processing required for implementation.
[0023] In a related aspect, a virtual reality (VR) therapy system is provided, for example, for providing psychotherapy to a patient. The VR therapy system may include VR hardware for providing a virtual environment, for example, as described above. The VR hardware may thus include at least a head mounted display unit having sound generation capabilities and a VR input device. The VR therapy system may further include a computer, for example a workstation, coupled to the VR hardware or a portion thereof.
[0024] The VR treatment system may be configured to implement the VR environment module to present a set of treatment space areas (scenarios) to the patient in the virtual environment using the VR system. Each treatment space area may include at least one patient activity (scenario) task to be performed by the patient in the virtual environment by interacting with the VR system. The patient may be represented in the virtual environment by a patient character, at least a portion of which may be visible at least part of the time. At least one of the patient activity tasks may include at least a dialogue task in which the patient character interacts with a task character in the virtual environment, or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment. At least part of the time, the virtual environment may include a representation of a coach character.
[0025] The VR therapeutic system may further be configured to implement a mental anxiety measurement module to determine a mental anxiety state parameter representative of the patient's mental anxiety state by measuring one or more characteristics of an interaction between the patient character and a task character or one or more objects within the virtual environment.
[0026] The VR therapy system may further be configured to implement a performance monitoring module to monitor the performance of patient activity tasks in the virtual environment.
[0027] The VR therapy system may further be configured to implement a detector module to detect one or both of non-performance of the patient activity task and a state of patient mental anxiety indicated by the mental anxiety state parameter.
[0028] The VR therapeutic system may further be configured to implement a patient feedback module to assist the patient in completing a patient activity task in the virtual environment by providing VR feedback including one or both of verbal and visual prompts in the virtual environment in response to detecting a state of patient anxiety using a coach character.
[0029] Further features of the VR therapeutic system may be as described above. For example, the VR hardware may include an eye-tracking device, and the mental anxiety measurement module may be coupled to the eye-tracking device to determine the mental anxiety state parameter from the patient's gaze direction. Similarly, the system may include a patient state neural network module having one or more inputs for receiving one or more characteristics of the interaction between the patient character and the task character or one or more objects, and an output for providing a mental anxiety state parameter representing the patient's mental anxiety state. Further details of an exemplary patient state neural network module are described below.
[0030] The VR therapy system may be partially implemented using software, for example, the modules described above may be implemented in software, i.e. instructions for a computer, or in hardware, e.g. ASIC (Application Specific Integrated Circuit), or a combination of software and hardware.
[0031] In a further related aspect, a method of controlling a virtual reality (VR) therapy system for providing psychotherapy to a patient is provided. The VR therapy system may include VR hardware to provide a virtual environment, the VR system including, for example, a head mounted display unit and a VR input device having sound generation capabilities, and one or more computers coupled to the VR hardware or a portion thereof. The method may include presenting a set of therapy scenarios to the patient in the virtual environment using the VR system, each therapy scenario including at least one scenario task to be performed by the patient in the virtual environment by interacting with the VR system. The patient may be represented in the virtual environment by at least a portion of a patient character. At least one of the scenario tasks may include at least a dialogue task in which the patient character interacts with a task character in the virtual environment, or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment. At least some of the time, the virtual environment may include a representation of a coach character.
[0032] The method may further include determining a mental anxiety state parameter representing the mental anxiety state of the patient by measuring one or more characteristics of an interaction between the patient character and the task character or one or more objects in the virtual environment. The method may further include monitoring performance of a scenario task in the virtual environment. The method may further include assisting the patient in completing the scenario task in the virtual environment in response to one or both of non-performance of the scenario task and the state of mental anxiety of the patient indicated by the mental anxiety state parameter, using a coach character to provide VR feedback including one or both of verbal and visual prompts in the virtual environment in response to detecting the state of mental anxiety of the patient.
[0033] Additionally, one or more storage media are provided that carry software, i.e., computer instructions, so as to (when executed) implement the systems and methods described above.
[0034] The instructions may be configured to implement the systems and methods on one or more computers, e.g., on a general-purpose computer system, and / or on a mobile device, and / or on a digital signal processor (DSP), and / or on configurable or dedicated hardware. The storage medium may comprise a carrier such as a disk or a programmed memory, e.g., a flash drive or other memory. The software, i.e., code and / or data for implementing the systems and methods, may comprise source, object, or executable code in a conventional programming language (interpreted or compiled), such as C, or assembly code, or code for a hardware description language. Such code and / or data may be distributed among several coupled components in communication with each other.
[0035] In a further aspect, a virtual reality (VR) therapeutic system for providing psychotherapy to a patient is provided, the VR therapeutic system comprising a computer system (e.g., one or more computers as described above), a head-mounted display unit for the patient connected to or including the computer system, and one or more sensors connected to the computer system for detecting patient activities performed by the patient. The computer system may be configured to cause the display unit to display a virtual environment for the patient character, the virtual environment including a task character in a therapeutic scenario and a coach character, either temporarily or permanently. The computer system may further be configured to cause the display unit to prompt the patient to perform a predefined activity in relation to the task character. The computer system may further be configured to determine, using the sensor, whether the patient has performed the predefined activity within a first period of time. The computer system may further be configured to cause the display unit to cause the coach character to provide an additional prompt for the patient to perform the predefined activity in response to determining that the patient has not performed the predefined activity within the first period of time.
[0036] The computer system may be further configured to determine, after the additional prompt, whether the patient has performed the predefined activity within a second time period. In response to determining that the patient has performed the predefined activity, the computer system may cause the display unit to progress the virtual environment to a different treatment scenario within the virtual environment or to a next level of the treatment scenario. In response to determining that the patient has not performed the predefined activity within the time period, the computer system may initiate an exit sequence to exit the patient from the virtual environment.
[0037] The virtual reality (VR) therapy system may further include a hand controller system that tracks the patient's hand movements, such as one or a pair of handheld controllers and / or one or more local or remote sensors that sense the position of a hard / controller. The predefined activities may include one or more of touching a virtual object, holding a virtual object, and moving a virtual object in the virtual environment. The sensor may include a microphone, and the predefined activities may include talking to a task character, and the computer system may detect when the patient is talking to the microphone to determine whether the patient has performed the predefined activities.
[0038] The computer system may further be configured to obtain one or more input parameters related to the patient's behavior, e.g., sensed input parameters, and then adjust the first period based on one or more of the input parameters. For example, the computer system may detect when and / or how fast and / or how loud the patient is speaking, and / or the computer system may detect the distance of the patient character in the virtual environment from the task character. In response, for example, if the input parameters related to the patient's behavior indicate that the patient is anxious or having difficulty performing the task, the first period may be shortened so that the coach character intervenes sooner. Similarly, the VR system may include an eye-tracking device attached to the display unit, and the input parameters may depend on the time spent by the patient looking at a predefined area of the virtual environment, e.g., looking down in the virtual environment. Additionally or alternatively, one or more of the input parameters, e.g., a parameter related to the patient's baseline anxiety, or a parameter related to a degree of severity of the patient's psychosomatic problem, or a parameter related to a type of scenario that the patient finds difficult, may be retrieved by the computer system from a remote data store.
[0039] In an implementation, the computer system is configured to cause the display unit to display an initial welcome virtual environment including the coach character but not the task character prior to displaying the treatment scenario virtual environment. In an implementation, the computer system is configured to cause the display unit to progress the virtual environment to a next level of the treatment scenario in response to determining that the patient has performed the predefined activity.
[0040] In some implementations, the coach character interacts with the patient character in the virtual environment by speaking to the patient character to provide additional prompts. The VR therapeutic system may further comprise a patient mobile device having a patient sensor input for sensing the patient's anxiety level and an audio output. The mobile device is configured to process data from the patient sensor input to monitor the patient's anxiety level when the patient is in the real world and away from the virtual environment, and to provide instructions to the patient in an audio output, e.g., the voice of the coach character, in response to the detected anxiety level. The patient mobile device may comprise or consist of a wearable (or portable) device for monitoring the patient's heart rate and / or galvanic skin response to sense the patient's anxiety level. The audio output may be provided in response to detection of an increased anxiety level, e.g., above a threshold and / or increasing above a threshold and / or having a pattern indicative of patient anxiety.
[0041] Some implementations of the VR therapeutic system collect patient data while the patient is in a therapeutic scenario within a virtual environment. The data may include a psychological anxiety state parameter representative of the patient's psychological anxiety state and / or one or more measurements or characteristics from which it is derived. For example, the patient data may include one or more of the aforementioned measurements, such as a proximity measurement, a gaze direction measurement, a patient verbal expression response time measurement, a task performance / non-performance measurement, a task duration (elapsed time) measurement, and a real-world biometric measurement. Additionally or alternatively, the patient data may include processed versions of such data, such as a difference in a biometric anxiety measurement, such as a cortisol level, heart rate, or GSR, between the beginning and end of the scenario. The patient data may be stored locally on one or more computers (computer systems) and / or remotely, such as on a remote server. In general, the connections between the components of the VR therapeutic system may comprise wired and / or wireless connections, and the data may be stored during or after the completion of a VR session or a particular scenario task in a VR session.
[0042] As mentioned above, some implementations of the VR treatment system include a patient (mobile) device, and such implementations may also include a remote server. The remote server may be configured to receive patient data collected from the computer system while the patient is in a treatment scenario in the virtual environment, the patient data representing, for example, the patient's performance in the virtual environment treatment scenario. The patient device may be configured to communicate with the remote server and output data to the patient depending on the patient's performance in the virtual environment treatment scenario, for example, to facilitate the patient's performance in a real-world scenario corresponding to the virtual environment treatment scenario.
[0043] The patient device may be configured to collect further patient data, e.g. corresponding to those described above, depending on the patient's performance in the real-world scenario corresponding to the virtual environment treatment scenario, and transmit this to the computer system via the remote server. The remote server may be configured to adjust the virtual environment treatment scenario in response to the further patient data, such that the virtual environment treatment scenario is adapted depending on the patient's performance in the real-world scenario.
[0044] Some implementations of the VR therapy system include a memory that stores data for displaying a plurality of different virtual environments for a plurality of different therapy scenarios. This memory or another memory may store a patient model associated with the patient, the patient model including one or more model parameters. The computer system may be configured to map the patient model associated with the patient to a subgroup of therapy scenarios based on the model parameters to present the subgroup of therapy scenarios to the patient. For example, the patient model may depend on the aforementioned patient data, such as cortisol levels, which may be used to select one or more of the therapy scenarios / tasks for the patient.
[0045] The computer system may also be configured to cause the head mounted display unit to present a subgroup of treatment scenarios to the patient to enable the patient to select a treatment scenario. Each of the subgroups of treatment scenarios may have a plurality of difficulty levels, and the computer system may be further configured to map the patient model to the difficulty levels that depend on the model parameters. The computer system may be further configured to receive information identifying patient activity within the virtual environment as input from one or more sensors, and to update model parameters of the patient model based on the input.
[0046] Some implementations of the VR therapy system are configured to use the collected (and stored) patient data to predict a therapy scenario and / or difficulty level for a patient, which may be the patient for whom the patient data was collected or may be a different patient.
[0047] For example, in one implementation, patient data related to a treatment scenario is processed to determine a progress or outcome measure for the patient, e.g., from differences between measurements of cortisol, heart rate, etc. before and after a treatment scenario. The progress or outcome measure may then be used to select a different treatment scenario or a different difficulty level of the same treatment scenario for presentation to the patient. Alternatively, a subgroup of such treatment scenarios or difficulties may be selected and presented to the patient so that the patient can select the next scenario / level.
[0048] Thus, in some implementations, the VR therapeutic system (and corresponding method) may include a cortisol sensor to sense the level of cortisol in the patient. The system / method may then determine the degree of performance of a scenario task or predefined activity, e.g., an outcome measure, depending on the level of cortisol in the patient. Additionally or alternatively, the system / method may determine the scenario task (e.g., type and / or level of a therapeutic scenario) or predefined activity for presentation in the virtual environment depending on the sensed level of cortisol in the patient.
[0049] In the same or another implementation, a machine learning classifier, such as a feed-forward or recurrent neural network classifier, may be used to process patient data for a patient to determine a classification output that includes data identifying one of a set of classes. Each class may, for example, correspond to a treatment scenario and / or difficulty level for the patient that is most likely to be beneficial to the patient. Additionally or alternatively, the classification output may be used to determine a set of treatment scenarios and / or difficulty levels for the patient. Additionally or alternatively, the classification output may be used to predict several treatment scenarios required by the patient until a certain value of a progress or outcome measure is achieved, e.g., indicating a successful final outcome. The prediction may be made for a new patient, e.g., by presenting the patient with test scenarios. In some implementations, the classification output may include a set of probability values or scores, one for each class, e.g., treatment scenario or set of treatment scenarios. In some implementations, the machine learning classifier may comprise a feed-forward classifier such as a deep neural network, e.g., a multi-layer perceptron (time-dependent patient data may be aggregated over time), or a recurrent neural network, i.e., a neural network having one or more recurrent neural network layers. The machine learning classifier, e.g., a neural network, may be trained using supervised learning techniques based on historical patient data and / or psychological measures of historical patient outcomes established using existing techniques.
[0050] Thus, in some implementations, a VR therapy system (and corresponding method) may be configured to collect patient data characterizing the patient's performance in scenario tasks or predefined activities. The system / method may then employ a machine learning subsystem configured to process the patient data to determine a type or level of therapy scenario (or set of these), a number of therapy scenarios to present to the patient in the virtual environment, or a level of therapy scenarios, e.g., an estimated number required for a successful end result.
[0051] Thus, in a further aspect, a virtual reality (VR) therapeutic system for providing psychotherapy to a patient is provided. The VR therapeutic system may comprise a VR system to provide a virtual environment, the VR system may include at least a head mounted display unit having a voice generating capability and a VR input device. The VR therapeutic system may further comprise one or more computers coupled to the VR system or a part of the VR system. The computer may be configured to present a set of therapeutic scenarios to the patient in the virtual environment using the VR system, each therapeutic scenario including at least one scenario task to be performed by the patient in the virtual environment by interacting with the VR system. The patient may be represented by a patient character in the virtual environment. At least one of the scenario tasks may include at least a dialogue task in which the patient character interacts with a task character in the virtual environment, or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment. At least some of the time, the virtual environment may include a representation of a coach character.
[0052] The computer may further be configured to implement a patient state neural network having a neural network input to receive sensor data constituting i) one or more characteristics of an interaction between the patient character and the task character or one or more objects in the virtual environment, or ii) an anxiety measurement, for example sensing one or more of the patient's heart rate, galvanic skin response, and cortisol level in the real world. The patient state neural network may process the neural network input to determine a mental anxiety state parameter representative of the patient's mental anxiety state.
[0053] The computer may be further configured to monitor performance of the scenario tasks in the virtual environment.
[0054] The computer may further be configured to assist the patient in completing the scenario task in the virtual environment in response to one or both of the non-performance of the scenario task and the state of mental anxiety of the patient indicated by the mental anxiety state parameter. For example, the VR system may assist the patient in completing the scenario task by providing VR feedback including one or both of verbal and visual prompts in the virtual environment using a coach character, for example, as described above. Additionally or alternatively, the VR system may assist the patient in completing the scenario task, for example, by modifying the treatment scenario online (i.e., during the selected scenario) to change the difficulty level of the treatment scenario, or by stopping the scenario and starting a new scenario / task. The treatment scenario may be modified online to reduce the scenario difficulty as described above, for example, by reducing the level of background noise, or the number of task characters in the scenario, etc.
[0055] In an implementation, the patient state neural network forms part of a closed loop feedback system (to the patient) since the patient state neural network (e.g. classification) output determines a mental anxiety state parameter that is used to control the coach character and / or scenario difficulty. The mental anxiety state parameter may thus be used to adjust or control the patient's experience in the VR environment, for example, to calm the patient by, for example, speech output, when patient anxiety is detected.
[0056] In an implementation, the patient state neural network may be as described above. For example, the patient state neural network may include one or more initial input neural network layers configured to receive neural network inputs and generate potential representations of the neural network inputs. The patient state neural network may further include at least one intermediate neural network layer having intermediate layer inputs coupled to the one or more input neural network layers and having intermediate layer outputs. The patient state neural network may further include one or more output layers having inputs coupled to the intermediate layer outputs to generate neural network outputs, for example. The neural network outputs may define a score distribution over a discrete set of possible patient anxiety values. The patient state neural network may be configured to generate a psychological anxiety state parameter from the score distribution. The psychological anxiety state parameter may be a categorical variable having two or more anxiety values (or having more anxiety stages), for example, defining an anxious state and a non-anxious state. The patient state neural network may then select a value, for example, the most likely anxiety value, according to the score distribution.
[0057] In a further aspect, the therapeutic VR system may include a subsystem, which may be referred to as an adaptive bio-behavior system, configured to adapt the course of a patient's treatment based on the patient's behavior and / or bio-responses while within the virtual environment.
[0058] In a virtual scenario, the scenario progresses based on a scenario-specific algorithm, also called a decision tree. The virtual scenario progresses down different branches of the decision tree based on the patient's activities in the virtual environment. For example, completing a scenario task may cause the virtual scenario to progress along a first branch of the decision tree, while non-completion of the task may cause the virtual scenario to progress along a second branch of the decision tree.
[0059] The adaptive biobehavior system may be configured to modify the decision tree / algorithm used to move the patient through a particular virtual scenario. That is, instead of just following a pre-set decision tree with a limited number of options using only such measurements, the adaptive biobehavior system may adapt the decision tree itself (e.g., by including more options or by introducing new endpoints). The adapted tree may then be used as the pre-set tree in the particular virtual scenario in subsequent therapy sessions. The adapted decision tree / algorithm may be used for future VR therapy systems and coach functions.
[0060] A particular virtual scenario can be configured to provide different stimulus intensities (e.g., depending on the number of virtual characters, the number of interactions with the virtual characters, and the type of interactions). The adaptive biobehavior system may be configured to adapt the decision tree based on the stimulus intensity.
[0061] In one implementation, the adaptive biobehavior system uses implicit metrics (e.g., reaction times and / or measures of conformance to expected responses to stimuli) measured while the patient is within the virtual environment to adapt the course of a treatment program provided to the patient by the therapeutic VR system within and / or outside of the VR environment.
[0062] Web and / or mobile data sources (e.g., user input on a smartphone during a VR session) are used to integrate with the therapeutic VR system to adapt the treatment scenario or stimulation intensity levels.
[0063] The system may be used in combination with other programs, interventions (e.g., behavioral and / or pharmacological), and other digital therapeutic techniques.
[0064] The patient's performance (or non-performance) of tasks within the VR environment may also be used to assess the patient's comprehension of therapeutic material (e.g., psychoeducation), which may be used to trigger further VR content based on the determined level of comprehension.
[0065] In one implementation, position and / or body tracking during the scenario task and during the event-specific stimulation are used as inputs to the adaptive biobehavior system, which can be used in algorithms to adapt to the patient's level of engagement and performance during the task or stimulation.
[0066] These and other aspects of the system are described with reference to the following drawings. [Brief description of the drawings]
[0067] [Figure 1] 1 shows a schematic diagram of a VR therapy system according to one embodiment. [Diagram 2] 1 is a flow diagram of a method performed by a VR system according to one embodiment. [Diagram 3] 1 shows a schematic diagram of a VR therapy system according to another embodiment. [Figure 4a] 1 illustrates a flow diagram of a method performed by a VR system according to one embodiment. [Figure 4b] 1 illustrates a flow diagram of a method performed by a VR system for determining a psychological anxiety state parameter, according to one embodiment. [Diagram 5] FIG. 1 shows a schematic diagram of a system for providing therapeutic treatment using a VR system and a mobile device, according to one embodiment. [Figure 6] 1 illustrates a flow diagram of a method performed by a VR therapy system according to one embodiment for providing a bus to a scenario. [Figure 7] 1 illustrates a flow diagram of a method performed by a VR therapeutic system according to an embodiment providing a cafe scenario. [Figure 8]1 illustrates a flow diagram of a method performed by a VR therapeutic system according to an embodiment for providing a store scenario. [Figure 9] 1 illustrates a flow diagram of a method performed by a VR therapy system according to one embodiment to properly position a patient before changing scenarios or levels. [Figure 10] 1 illustrates a flow diagram of a method performed by a VR therapeutic system according to one embodiment for providing a virtual welcoming room. [Figure 11] 1 illustrates a flow diagram of a method performed by a VR therapeutic system according to one embodiment in which a patient's anxiety rating is determined. [Figure 12] 13 shows a flow diagram of a method performed by a VR therapeutic system according to another embodiment in which a patient's anxiety rating is determined. [Figure 13] FIG. 1 is a schematic diagram of a clinical platform including a VR platform for treating multiple mental health conditions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0068] In the drawings, like elements are indicated by like reference numbers. Detailed Description FIG. 1 shows a schematic diagram of an example of a virtual reality (VR) therapeutic system 1 (referred to herein simply as a “VR system”) comprising a computer system 2 and VR hardware 3 including a wearable display unit 4 and an input device 5 for a patient.
[0069] A computer system is configured to connect to the wearable display unit and cause the display unit to display the virtual environment for the patient character. The computer system may be a stand-alone workstation, a computer network, or may be a small computer integrated into the wearable display unit or otherwise carried by the patient.
[0070] 2 is a flow diagram showing steps of a method for providing a therapy session using a VR system comprising a computer. The computer is configured to display a virtual environment for a patient character (step S1), prompt the patient to perform a predefined activity (step S2), determine whether the patient has performed the predefined activity (step S3), and in response to determining that the patient has not performed the predefined activity, provide an additional prompt for the patient to perform the predefined activity (step S4). Optionally, the computer is further configured to determine whether the patient has performed the predefined activity after providing the additional prompt (step S5), and in response to determining that the patient has performed the predefined activity, advance the virtual environment (step S6a) or initiate an exit sequence (S6b) in response to determining that the patient has not performed the predefined activity.
[0071] To realize a VR therapeutic session using the VR system 1, the system uses the programming of 3D environment and interaction. The software used to develop a VR session can be divided into two main categories: a) 3D modeling and animation software, and b) development engines. Within category a) (3D modeling and animation software), three-dimensional objects are generated by using techniques of CAD software, as well as visual aspects of surfaces, ambient lighting, natural effects, dynamic effects (forces, gravity, etc.), and all kinds of animations. This type of software is also used in the production of movies, video games, and projects. 3D modeling software that can be used includes Blender (trademark pending), Autodesk 3DStudio Max (trademark pending), and Autodesk Maya (trademark pending). Regarding the second category b), different names are used to describe it, including development engines, video game engines, and graphic engines (which generate the interactive images used in video games or VR applications). The word "engine" refers to a program or part thereof that performs a certain task. The development engine provides a set of basic programmed functions that are common in all VR applications, such as: a) a rendering engine that generates 2D and 3D graphics; b) a collision detection engine; c) possible interaction with the environment; d) sound and music; e) animation; f) artificial intelligence; g) communication with the network; h) memory management. Examples of graphics engines that may be used include Uinty3D (trademark pending) and Unreal Engine (trademark pending).
[0072] The development of a VR session can also be divided into three categories: a) environment, b) character, and c) scripting.
[0073] a) Environment: This contains the passive elements of the scenario, everything displayed will remain static.
[0074] b1) Character Creation: Creating the characters and all the underlying logic to puppeteer them.
[0075] b2) Character Animation. This is the process of animating the characters created in (b1) to move. Character animators can use Autodesk Maya and Autodesk MotionBuilder. Animation can be created using a motion capture system such as Optittrack.
[0076] c) Scripting: Using a game engine (e.g. Unity) to assemble all the elements, set their positions, and program their logic and behavior.
[0077] Each scenario will have the same environment model. The cafe is the same cafe for all levels 1-5. The characters in each level may vary, some characters are in some levels and not in others.
[0078] FIG. 3 shows a schematic diagram of an example of a VR therapy system 1 comprising a workstation 6, a display unit 4 with audio capabilities, one or more input devices 5, and a VR hardware 3 including a number of sensors 7 and 8. One of the sensors 7 in the display unit is an eye-tracking sensor for measuring the gaze direction of a patient character in a virtual environment. Another sensor 7 in the display unit is a position tracking sensor for tracking the patient's position. Other sensors 8 outside the display unit include a tracking and input sensor attached to a handheld controller (one of the input devices 5) for tracking the patient's hand movements in the virtual environment. A biosensor on the handheld controller can be used to measure sweat / cortisol levels, which can be used to assess the patient's anxiety level during a therapy session. The sensors 7 and 8 also include a microphone for voice detection. The VR system 1 further comprises a number of modules including a VR environment module 9, a measurement module 10, a monitoring module 11, a detector module 12, a feedback module 13, a predictor module 14, and a patient state neural network module 15. The VR system is configured to implement a VR environment module 9 to present a set of treatment spatial regions to the patient in a virtual environment using the VR system 1. These spatial regions correspond to different treatment scenarios. The VR system 1 is further configured to implement a measurement module 10 to determine (together with a neural network module 15) a mental anxiety state parameter representative of the patient's mental anxiety state, a monitoring module 11 to monitor the performance of the patient activity task, a detector module 12 to detect non-performance of the task and / or the patient's mental anxiety state, and a feedback module 13 to assist the patient in completing the task. The VR system 1 further comprises a data storage 16 for storing data related to the patient and the virtual environment, which is accessible by the VR environment module 9 and the workstation 6 to render the virtual environment.The VR system 1 is configured to implement a predictor module 14 to predict the number or spatial domains (scenarios) and difficulty of sessions that the patient is recommended to pursue. The predictor module 14 preferably uses machine learning to provide the predictions. The workstation 6, modules (9-15) and data storage 16 may form the computer system 2 of the VR system of FIG.
[0079] Figure 4a is a flow diagram illustrating steps of a method for providing a therapy session via a VR system, such as the VR system shown in Figure 1 or Figure 3. The method includes the steps of presenting a therapy scenario including a scenario task (step S7), invoking a determination of a psychological anxiety state parameter (step S8), monitoring the performance of the scenario task (step S9), and assisting the patient in completing the scenario task with a coach character (step S10). The psychological anxiety state parameter can be determined from a range of different parameters related to the patient character's interaction with the virtual environment.
[0080] FIG. 4b is a flow diagram illustrating an example of a VR system invoking the determination of a mental anxiety state parameter (step S8). The VR system can invoke the measurement module 10 to measure the gaze direction of the patient character (step S8a), the proximity of the patient character to objects or other characters in the virtual environment (step S8b), the time it takes the patient character to respond to verbal prompts (step S8c), the time it takes the patient character to respond in relation to objects in the virtual environment (step S8d), and optionally the patient's physical / biological response (e.g., cortisol levels) (step S8e). One or more of these measurements are used to determine or adjust the patient's mental anxiety state parameter (step S8f), which is then used by the feedback module 13 to adjust the assistance provided to the patient character. In general, if the state parameter indicates a high level of anxiety, the feedback module is implemented to provide prompts or encouragement from the coach character more quickly. Any one of the measurements can be invoked in response to an event in the scenario (e.g., the movement of other characters or objects in the environment) or at a given time interval (e.g., every second). Proximity measurements may be triggered by movement of the patient character or by movement of objects or other characters in the virtual environment. Eye tracking may be triggered by the patient character moving in front of the task character in the scenario and can be used to determine whether the patient character maintains eye contact with the task character.
[0081] In one example, the neural network module 15 is used to determine the mental anxiety state parameter. Inputs to the neural network module 15 include one or more measurements in the virtual environment from the measurement module 10 (steps S8a-S8d) and / or physical measurements from the sensor 8 (step S8e). The neural network 15 may directly output a mental anxiety state parameter, or a set of scores or probability values, one for each of a set of classification categories that may represent the degree of mental anxiety of the patient. These scores may then be used to classify the patient as anxious or non-anxious and / or grade the patient's anxiety into more than two categories.
[0082] The VR system 1 utilizes techniques from cognitive behavioral therapy (CBT), an evidence-based treatment for several mental health conditions. It is an interactive therapy that allows patients to test their automatic beliefs and emotions in a virtual environment in order to modify the patient's reaction to those automatic responses.
[0083] In one example, the VR system 1 has access to a number of different treatment scenarios that it can present to the patient in a virtual environment. The VR system is configured to provide the patient with several levels of carefully graded difficulty to test the patient's responses and behavior. The predictor module 14 can be used to select a subgroup of scenarios to present to a particular patient. For example, to treat social avoidance, the system can use different scenarios including a cafe, a store, a pub, a doctor's waiting room, a bus, and a street. Each scenario is associated with a task / activity to be completed by the patient's virtual character and a difficulty level that determines certain aspects of the scenario.
[0084] FIG. 5 shows a schematic diagram of a system comprising a VR treatment system 1 and a mobile device 17 with a mobile application for providing further treatment and collecting patient data between VR sessions. The mobile device 17 is connected to a wristband 18, which can be worn by the patient and includes sensors for collecting patient data (e.g., heart rate, GSR, cortisol / sweat levels) that can be transmitted to the mobile application. The mobile device 17 and the VR system 1 can communicate directly or through a server 20 (e.g., a cloud-based server) via a network 19 to share data. The mobile application can use the patient data from the wristband 18 to determine an anxiety rating for the patient and provide feedback based on the anxiety rating. For example, if the mobile application determines that the patient has a high anxiety level, the mobile application may provide verbal encouragement from the mobile device. Preferably, the voice providing the verbal encouragement is the same voice as the coach character in the virtual environment. The mobile device 17 or wristband 18 can track the patient's location using GPS tracking and provide the location to the application. The application can use the location to provide targeted feedback to the patient. For example, a mobile application may use the location to identify when the patient is in a real-world scenario that corresponds to one of the treatment scenarios in the virtual environment and provide appropriate feedback for that situation.
[0085] Thus, the patient mobile device may use the monitored patient activity / anxiety to identify when the patient is in a stressful real-world situation. In response, the mobile device 17 may then provide real-world feedback to the patient, e.g., a spoken audio output via headphones with the voice that the coach character uses to assist the patient in the virtual environment. The voice may provide soothing words and / or spoken instructions or encouragement from the mobile device 17 to the patient.
[0086] In one example, the mobile application collects information and feedback provided by the patient and transmits the patient data to the VR system 1. The VR system 1 modifies the VR therapy session for the patient using the patient data provided by the mobile application. Similarly, patient data collected by the computer system during the VR session (e.g., the patient's progress, achieved levels, and learning) can be transmitted from the VR system to the mobile application. Between VR sessions, the mobile application can use information provided by the VR system 1 to reinforce learning points and encourage the patient to practice the skills and behaviors learned in the VR session.
[0087] In one example, an automated CBT program is provided to the patient from the VR system 1. Time spent in the scenario, answers to questions, levels achieved, and goals are collected while the user is undergoing VR therapy. Patient data may be stored locally on the computer system of the VR system 1 and / or remotely on a remote server 20. Data is passed to the server 20 or directly to the mobile device 17 for processing. In general, connections between components of the VR therapy system may comprise wired and / or wireless connections, and data may be stored during or after the completion of a scenario task.
[0088] Between sessions, the mobile application uses this data to reinforce learning points and prompt the user to apply the learning in the real world. The mobile application provides an interface for the patient to give feedback, records whether the patient tried to apply the learning points in the real situation, and provides an evaluation. This feedback can be sent back to the server 20 or directly to the VR system 1. In the next VR session, the automated therapy is adjusted using the processed feedback to tailor the therapy to the user's needs.
[0089] Each scenario may have five difficulty levels, in which the VR system is configured as follows: Cafe Scenario 1. Give a queue and prompt the patient to order a drink. 2. More background characters (busier cafes), giving queues and encouraging patients to order drinks. 3. Give the patient a queue and prompt them to order a drink. Give the patient character a barista character who asks them to repeat the order. 4. Provide a customer character who leaves a wallet on the table and prompt the patient to warn the customer character. 5. Provide a child character blowing bubbles into the air and encourage the patient to pop the bubbles. Provide a further background character that watches the patient character as it performs the task.
[0090] Bus Scenario 1. The patient is invited to wait at the bus stop and then is invited to pay the fare and board the bus. 2. More background characters (more crowded). The patient is prompted to wait at the bus stop, then pay the fare and board the bus. 3. More background characters (more crowded). The patient is prompted to wait at the bus stop, then pay the fare and board the bus. 4. Have a passenger character ask the patient character to ring the bell, with other passenger characters looking at the patient character. 5. Provide more background character (very crowded). The patient is prompted to wait at the bus stop, then pay the fare and board the bus.
[0091] Street Scenario 1. The patient is asked to leave the front door and wait for a taxi on the street. 2. More background characters (more crowded). Patients are encouraged to leave the front door and wait for a taxi on the street. 3. More background characters (more crowded). Patients are encouraged to leave the front door and wait for a taxi on the street. 4. The patient is encouraged to wash the graffiti off the wall with their back to the street, away from the front door. 5. More background characters (very busy). Encourage patients to leave the front door and wait for a taxi on the street.
[0092] Doctor's waiting room (GP) scenario 1. Encourage patients to wait in queue. 2. Encourage the patient to wait and give their personal information to the receptionist. 3. Blow virtual flyers into the air and encourage the patient to grab them, with the other characters looking at the patient character. 4. Provide a slightly angry human character asking the patient character to hand over the pen holder. 5. The patient is prompted to wait to be called by the doctor and ask the receptionist when it is his / her turn.
[0093] Pub Scenario 1. The patient is prompted to wait at the pub for a friend character to arrive. 2. More background characters (busier): The patient is prompted to wait in a pub for a friend character to arrive. 3. Provide more background characters (more crowded), including loud sports fans. The patient is encouraged to wait in a pub for a friend character to arrive. 4. Have a sports fan character ask the patient character where the bathroom is. 5. With the other characters watching the patient character, have the barman character ask the patient character to ring the bell for last orders.
[0094] Store Scenario 1. While the store owner character is looking at the patient character, he or she encourages the patient to hang out in the store. 2. Provide other customer characters in the store. 3. Ask the patient to turn his or her back to the store and pick up three items from the shelf. 4. More background characters (more crowded). Ask the patient to turn his / her back to the store and pick up three items from the shelf. 5. Provide more background characters (very busy). Have the patient wait at the register while the storekeeper character enters the patient character's items into the register.
[0095] Difficulty may be further adjusted by adjusting the amount of time a user is given at a given level of a scenario. Difficulty may also be adjusted by modifying the behavior of background characters, for example by adjusting the frequency and / or duration that background characters view patient characters.
[0096] Each time an activity is required by the patient, the VR system determines whether the patient has successfully completed the activity within a given period of time and responds accordingly. Prompts occur in the context of the scenario and may be delivered verbally by one of the characters in the situation and / or by a coach character. Prompts may also include drawing the patient's attention to a particular object required to complete a level. The system sets a fixed number of prompts (typically at least two) before the system determines that the patient will not complete the task, and in response to that determination, ends the scenario. If the patient does not successfully complete a level, the system automatically (after a period of time since the last prompt) exits the patient from the scenario, so that the patient does not remain in that level indefinitely. To exit the patient character from the scenario, the system triggers an "outro" sequence, and the patient is not given the opportunity to proceed and must repeat the level to progress. This is due to the careful grading of difficulty in the program to maximize learning and ensure therapeutic benefits. This interaction is important to the key concept of "presence" in a VR environment, which relates to the sense that the patient "really is there" and believes and behaves as if they are part of the represented virtual world. When the program's characters, environments, and structures do not behave believably, the patient may have a disruption of presence, which limits the effectiveness of VR therapy.
[0097] Once difficult situations are coached and progress is made, the patient can move on. Pacing of the therapy is key to its effectiveness, and the prompts ensure that only patients who are ready to move on by successfully completing a level do so. Because the dialogue is conducted by an in situ character and coach, it reflects realism by placing the patient in a situation that the patient would actually be in, thus ensuring that the therapeutic benefits are particularly relevant and therefore resulting in better outcomes. The dialogue also creates a degree of familiarity between the patient and the coach character who is relied upon for prompts whenever present. This is useful because the patient can expect to get encouragement in testing their beliefs in particularly difficult situations within VR, and know that they will be prompted if they do not complete an activity immediately.
[0098] Known VR CBT products do not have any dialogue in the virtual world. This means there is limited presence because the practitioner or clinician in the room with the patient is often speaking while the patient is in the virtual environment. Many programs do not feature any activity, but simply involve exposure to threatening situations. There is no ability to test new responses to automatic thoughts and emotions.
[0099] The VR system can provide different objects in a virtual environment that the patient can interact with using a hand controller input device or a microphone to make selections and complete tasks. Sphere Selector. The patient character can reach and grab a sphere associated with their choice (to select a scenario, a sphere is displayed beneath a floating 2D image representing the scenario). Once grabbed, the sphere will burst or otherwise give an indication to confirm the choice. The sphere selector can allow the patient to select which scenario they want to do, at what level to start, whether they feel safer (yes / no), or if they are ready with a single choice. Sphere Slider. The coach can ask the patient how confident they are entering different situations. The patient can indicate their confidence level by reaching and grabbing a sphere in front of them and moving it along a number line (e.g., marked 1-10). Such indications are recorded throughout the course of the session. · Slide a pen holder. The patient character can reach a pen holder and slide it towards another character at level 4 in the doctor's waiting room. Glowing footprints. Patients may view glowing footprints as they are prompted to move to the correct location. Illuminating object prompts. Certain objects may illuminate when the patient needs to interact with them. Carryable objects. Carryable objects are picked up by squeezing the trigger on the controller when the patient's hand is near the object, and dropped by releasing the trigger. Certain objects such as hoses (Street Level 4) do not fall when the trigger is released and can still be passed from hand to hand. Certain objects such as payment cards (in the Bus and Shop scenarios) are attached to the patient's hand at the start of the level. If the object has a trigger activity (such as a hose water stream), it is controlled by how hard the patient squeezes the trigger. Voice detection. The patient will be required to speak - it may not be necessary to implement voice recognition, instead it may simply be possible to detect whether the patient has spoken or not.
[0100] An example of physical, non-verbal interaction is given in a bus scenario, where the patient must pay every time they board the bus; if they do not, the coach (if present) will prompt the patient to pay. The payment machine also visually lights up to catch the patient's attention and indicate the required payment activity.
[0101] FIG. 6 is a flow diagram showing some of the steps performed by the VR system in a therapy session involving a bus scenario. The patient character loads a virtual payment card in his / her hand in front of the bus driver. The VR system determines whether the patient makes a payment to the driver based on input from a hand controller that tracks the movement of the hand holding the virtual payment card. If the system determines that the patient character has made the payment, the coach gives the patient character positive feedback (encouragement) and encourages him / her to take a step forward. If the system determines that the payment has not been made within a 15 second period, the system provides a visual prompt followed by a verbal prompt through the coach character by telling the patient character how to make the payment. The system determines whether the patient character performs the required task within a 10 second period and if not, exits the scenario.
[0102] FIG. 7 is a flow diagram illustrating some of the steps performed by the VR system when executing the cafe scenario. After the patient character waits his or her turn in the queue, the coach provides verbal encouragement to the patient and asks if the patient knows what he or she wants to drink. The barista character asks the patient character to order a beverage. This prompts the patient character to move forward and out of the queue to order a drink from the barista character. Position tracking and voice recognition are used to determine if the patient completes the task. In response to completing the task, the barista character responds to the patient character and the patient can proceed to the next level. If the patient character does not move forward and no speech is detected within a 15 second period, the VR system determines that the patient has not completed the task. In response to determining non-competition, the coach provides encouragement to the patient character and the barista provides an additional prompt to order. Again, in response to determining that the patient has completed the task, the VR system moves forward, the barista responds to the patient character, and the patient can proceed. If the patient does not order within 15 seconds of the second prompt from the barista, the system determines that the patient is not on task and ends the scenario.
[0103] FIG. 8 is a flow diagram illustrating some of the steps performed by the VR system in a therapy session involving a store scenario, where the predefined activities the patient is prompted to complete include selecting three items from a list and placing them in a basket in a store. The coach provides instructions to the patient, and a first item on the floating list appears. The VR system determines through input from the hand controller whether the patient character picks up the item. In response to the patient character picking up the item, the VR system provides the second item on the list, and the process is repeated. In response to the VR system determining that the patient has not completed the task within a 30-second period, the system provides a visual prompt related to the item, and after another 10 seconds, provides a prompt through the coach. If the patient completes the task in response to these prompts, the VR system advances the scenario to the next item. If the VR system determines that the task is not completed after providing additional prompts, the scenario is terminated.
[0104] When moving between scenarios or different levels of a scenario, the patient will have to reposition himself / herself to a starting point that may differ from the previous starting point, and the VR system is configured to prompt the patient to move to the correct position and to determine when the patient character is in the correct position.
[0105] 9 is a flow diagram showing steps performed by the VR system to reposition the patient before starting a new scenario or a new level. The VR system provides footprints on the floor / ground of the virtual environment to indicate the correct position. The system determines whether the patient character is standing on the footprints by tracking the patient's position relative to the virtual environment. If the patient is not in the correct position after a 10 second period, the coach gives a verbal prompt to move to the correct position. In response to determining that the patient character is standing on the footprints, the footprints are removed and the VR system proceeds to display the next level or scenario.
[0106] Prior to starting a particular scenario (bus, cafe, store, etc.), the VR system can provide a virtual environment including a "welcome room" where the coach can be introduced to the patient and, in some examples, the patient can select a scenario for the therapy session. FIG. 10 is a flow diagram illustrating steps associated with the welcome room performed by the VR system. The patient character enters the welcome room. The coach explains how the patient character can interact with the virtual environment. A virtual object (e.g., a sphere selector) is provided with which the patient character can interact. The coach asks the patient to indicate a confidence level before selecting a scenario for the therapy session.
[0107] The coach's interaction within the virtual environment is provided in response to the system determining whether the patient has successfully completed an activity. This determination can include measuring patient input associated with a particular activity (verbal, physical, or both) required within a specified time period. The particular time period for a given task can be based on careful study of what is appropriate and expected for a given activity, which leads to verbal and visual prompts in case of failure to allow the patient to complete a given task and move on to the next level / scenario.
[0108] This period can preferably be determined or adjusted on the fly. For example, in one example of a VR system where the display device is equipped with eye-tracking capabilities, this period can be determined based on the patient character's gaze on the virtual environment. If the patient character looks down at the floor when asked to order coffee, the computer system can set a shorter period to provide another prompt more quickly. Conversely, if the patient character maintains eye contact with the task character (e.g., the barista in a café scenario), this period may be extended.
[0109] In another example, a proximity sensor is used to determine the distance of the patient character to an object or virtual character in the scenario, and the period is determined or adjusted based on the distance. For example, if the patient character moves closer to the task character (e.g., moving forward toward the ballista), the period may be extended to give the patient more time to successfully complete the task before a prompt is given. If it is determined that the patient character is too far away or has moved too far, the period can be shortened to give a prompt more quickly.
[0110] This period may generally be based on the patient's psychological anxiety state parameters, which in turn may be based on gaze direction and / or proximity as well as other measurements.
[0111] FIG. 11 is a flow diagram illustrating how the VR system or patient mobile device can determine whether to give further prompts to the patient in a scenario or real-world situation based on the anxiety rating determined from the patient's biological response. The anxiety rating associated with the patient is stored. The anxiety rating is set as a normalized anxiety rate based on the patient's measured heart rate and skin conductance (GSR) and / or cortisol level. The scenario or situation has an associated task activity, e.g., ordering a coffee, for the patient or patient character to complete. The coach asks the patient to perform the task activity. After x seconds, a new anxiety rating is calculated. The new anxiety rating is compared to the normalized anxiety rating to determine an action. If the anxiety rating increases beyond a threshold, further prompts / encouragement are given to the patient. If the state parameters do not indicate a significant increase in the anxiety rating, no action is taken. After determining the new anxiety rating, this value can be set as the new normalized anxiety rating. This process can be repeated until the required task activity is performed or a pre-set number of prompts are provided. In a particular example, the anxiety rating is updated every second (i.e., x=1) to quickly determine if the patient is becoming more anxious, whereby the anxiety rating is used to dynamically set the period between prompts provided as a function of the patient's anxiety.
[0112] In another example, instead of or in addition to calculating an anxiety rating based on heart rate and skin conductance when in a VR session, the anxiety rating can be based on the patient character's activity in the virtual environment, for example, the anxiety rating can be based on the patient character's gaze using eye tracking.
[0113] 12 is a flow diagram illustrating a more advanced method of determining when to provide a prompt to a patient character based on mental anxiety state parameters. Instead of simply comparing the current anxiety rating to a normalized anxiety rating, the normalized anxiety rating history is compared to a predefined pattern. The pattern may be based on data from previous sessions with the patient. Depending on whether the anxiety history matches the predefined pattern, a coach can be used to provide appropriate feedback (e.g., further prompts to complete a task activity).
[0114] Cortisol / sweat levels can be measured by including a wearable sweat biosensor. Users with anxiety tend to sweat on the VR hand controllers, and the biosensor can be included as a sticker that is attached to the hand controller. Cortisol / sweat levels can also be used as a predictor of the need for treatment (e.g., to determine appropriate scenarios / levels). It can also be used to provide an objective outcome measure of treatment by comparing cortisol / sweat levels before and after completion of a treatment session.
[0115] During a session, the following information may be recorded: General information: Participant ID, session number, date and time, application build. Tracking Data: Head position and orientation, hand position and orientation, controller button presses, controller button releases. -Scenario Events: Scenario start / end, and scenario pause / resume. User input / activities / interactions with spheres and other similar props, as well as audio where applicable: initial calibration (countdown), confidence, anxiety level and similar questions requiring input from the participant, session options start / continue / end, position reset, walk to footprint.
[0116] Activities of the patient character within the VR environment that the VR system can detect include: · Cafe Level 5 - Popping bubbles. · Store Levels 3 and 4 - Pick up objects and drop objects. Bus levels 1-5 - card payment. Bus Level 4 – Press the bus stop button. ·GP Level 3 - Grab flyers, drop flyers. ·GP Level 4-Hand over the pen. ·GP Level 5-Tell the receptionist that you've been waiting a while? Pub level 4 - point me in the direction of the toilets? Pub Level 5 - Ring the bell for last orders. Street Level 4 - Press the trigger to spray water, then release the trigger.
[0117] The dialogue is designed to allow for significant incorporation of learning into the experiment using ideas about automatic responses to produce therapeutically relevant change. Prompts are particularly effective because they are designed at critical points in the treatment where difficulty and anxiety are anticipated, and are delivered in a variety of ways (by the coach, by other virtual characters, verbally, and visually) to increase the chances of successful learning.
[0118] Raw data can be collected during a VR session and processed by a computer system to determine more useful data structures, such as: -How long did it take the patient character to answer the questions? -How long did it take the patient character to perform the activity? -How confident was the patient based on the model of upper limb movement when performing the activity? -How much time did the patient character spend looking down at the floor? -Did the patient character look directly into the eyes of another character? How often? -How far was the patient character from the virtual coach? Did that distance change during the session? -Did the patient character press the right button? (There may be delays just because they are not used to using the hand controller.)
[0119] A computer system can use such data structures as input and, based on the patient data, determine the correct treatment scenario and difficulty level to provide treatment to that patient, for example using neural networks and predictor modules. As more data is collected over time, the VR system can be used to predict how many sessions or which scenarios / difficulty levels a participant will be recommended to pursue. This can be useful to help the service manage patient treatment, treatment resources (allocation to bring in more patients), estimate how many headsets will be needed, costs, etc. Machine learning can be used with the data to improve these predictions.
[0120] Patients and health services typically think in terms of specific and unique mental disorders: depression, for example, or generalized anxiety disorder. However, people with one disorder are likely to meet the criteria for other disorders, possibly even three or more. It is also clear that psychogenic problems are dimensional; that is, they exist on a spectrum of severity. Depression, for example, is less of a discrete category of experience than a relatively severe case on the spectrum of low mood. Most people will find themselves at different points on this spectrum over their lifetime. Furthermore, many experts consider mental disorders to be complex combinations of psychogenic problems that are themselves dimensional, rather than separate disorders.
[0121] The above factors mean that there is a significant range of transdiagnostic treatment techniques to complement disorder-specific interventions. The example VR system described herein can make such treatments available. For example, the VR system can realize different scenarios to help the user become more active, overcome worries, understand the impact thoughts can have on emotions, and engage in daily activities. These scenarios present evidence-based cognitive and activity treatment techniques. They are therefore likely to be of great value to people with a wide range of psychogenic problems, including the most common disorders depression and anxiety. The inventors pursue a modular approach to treatment delivery. The VR system provides a VR platform that includes a wide range of VR scenarios, each of which focuses on a specific treatment technique, which may be supported by mobile applications and third-party treatment tools. Many of these VR scenarios or modules will be transdiagnostic. Others will focus on specific disorders (e.g., OCD). Specific combinations of modules can be tested to establish efficacy for specific diagnoses. The VR platform can provide a dashboard where clinicians and patients can select the modules they consider most effective for them.
[0122] Figure 13 shows a schematic of how a VR platform could be part of a general clinical platform for treating multiple mental health conditions. The VR platform provides access to different scenarios (scenarios A-E) that can be used to treat one or more disorders.
[0123] Many alternatives will occur to those skilled in the art, and the invention is not limited to the described embodiments, but encompasses modifications apparent to those skilled in the art that are within the spirit and scope of the appended claims.
Claims
1. 1. A virtual reality (VR) therapy system for providing psychotherapy to a patient, comprising: The VR therapeutic system further comprises a VR system for providing a virtual environment, the VR system including at least a head mounted display unit having a sound generating capability and a VR input device. one or more computers coupled to the VR system or a part thereof, the one or more computers comprising: configured to present a set of treatment scenarios to the patient within the virtual environment using the VR system, each treatment scenario including at least one scenario task to be accomplished by the patient within the virtual environment by interacting with the VR system; the patient is represented in the virtual environment by a patient character, and at least one of the scenario tasks includes at least a dialogue task in which the patient character interacts with a task character in the virtual environment or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment, and at least a portion of the time, the virtual environment includes a representation of a coach character, and the one or more computers further include: determining a psychological anxiety state parameter representative of a psychological anxiety state of the patient by measuring one or more characteristics of an interaction between the patient character and the task character or the one or more objects in the virtual environment; monitoring performance of the scenario tasks in the virtual environment; In response to one or both of the non-performance of the scenario task and the state of mental anxiety of the patient indicated by the mental anxiety state parameter, A VR therapeutic system configured to assist the patient in completing the scenario task in the virtual environment by providing VR feedback including one or both of verbal and visual prompts in the virtual environment using the coach character.
2. Determining the psychological anxiety state parameter includes determining, in the virtual environment, a gaze direction, a proximity of the patient character to the task character, a time to respond to a verbal expression of the task character, and a time to perform an activity in relation to the one or more objects. The VR therapy system of claim 1 , comprising measuring one or more of:
3. The VR therapy system of claim 1 or 2, further comprising a patient state neural network having one or more inputs for receiving one or more characteristics of the interaction between the patient character and the task character or the one or more objects, and an output for providing the mental anxiety state parameter representing the mental anxiety state of the patient.
4. 4. The VR therapy system of claim 1, further comprising a patient mobile device configured to: monitor one or both of a patient activity level and a patient anxiety level when the patient is in the real world and away from the virtual environment; identify when the patient is in a stressful real-world situation in response to said monitoring; and provide real-world feedback when the patient is in the real world and away from the virtual environment in response to said identification; wherein the real-world feedback includes speech audio output to the patient in a voice of the coach character in the virtual environment.
5. 5. The VR therapy system of claim 4, wherein the one or more computers are further configured to capture patient data from the patient while the patient is in the therapy scenario in the virtual environment, and the patient mobile device is configured to provide the real-world feedback using the captured data.
6. 6. The VR therapeutic system of claim 4 or 5, wherein the patient mobile device comprises or consists of a wearable device for monitoring the patient's heart rate and / or galvanic skin response and / or cortisol level in the real world and away from the virtual environment to determine the degree of the patient anxiety.
7. The VR treatment system of any one of claims 1 to 6, wherein the set of treatment scenarios includes a set of different treatment scenarios, each treatment scenario configured to simulate a real-world scenario that the patient finds difficult, each treatment scenario having a set of difficulty levels, and the one or more computers are further configured to provide a welcome room VR scenario including the coach character, in which the patient character can select one of the treatment scenarios and a difficulty level for the selected treatment scenario, the difficulty level being equal to or greater than a previously completed difficulty level for the treatment scenario.
8. 8. The VR therapy system of claim 7, wherein the one or more computers are further configured to modify the selected therapy scenario to change the difficulty level by modifying one or more of a background noise level, a number of task characters in a selected therapy scenario, a duration of the selected therapy scenario, a position of the task characters in the virtual environment, and a behavior of the task characters.
9. 1. A virtual reality (VR) therapy system for providing psychotherapy to a patient, comprising: The VR therapeutic system further comprises: VR hardware for providing a virtual environment, the VR hardware including at least a head mounted display unit having a sound generating capability and a VR input device; The VR therapy system includes a computer coupled to the VR hardware or a portion thereof, and implementing a VR environment module to present a set of treatment space areas to the patient in the virtual environment using a VR system, each treatment space area representing at least one treatment to be performed by the patient in the virtual environment by interacting with the VR system. and at least one patient activity task, the patient being represented by a patient character within the virtual environment, at least one of the patient activity tasks including at least a dialogue task in which the patient character interacts with a task character within the virtual environment or a non-verbal activity task in which the patient character performs non-verbal activities with respect to one or more objects within the virtual environment, the virtual environment including a representation of a coach character at least some of the time, the VR therapeutic system further comprising: a mental anxiety measurement module for determining a mental anxiety state parameter representative of a mental anxiety state of the patient by measuring one or more characteristics of an interaction between the patient character and the task character or the one or more objects in the virtual environment; a performance monitoring module for monitoring performance of the patient activity task in the virtual environment; a detector module for detecting one or both of non-performance of the patient activity task and a state of patient anxiety as indicated by the anxiety state parameter; and a patient feedback module to assist the patient in completing the patient activity task in the virtual environment by providing VR feedback including one or both of verbal and visual prompts in the virtual environment in response to detecting a state of mental anxiety of the patient using the coach character.
10. 1. A computer-implemented method for controlling a virtual reality (VR) therapy system for providing psychotherapy to a patient, the VR therapy system comprising VR hardware for providing a virtual environment, the VR therapy system including at least a head mounted display unit having sound generation capabilities and a VR input device, and one or more computers coupled to the VR hardware or a portion thereof, the method comprising: presenting a set of treatment scenarios to the patient in the virtual environment using the VR therapeutic system, each treatment scenario including at least one scenario task to be accomplished by the patient in the virtual environment by interacting with the VR therapeutic system; the patient is represented in the virtual environment by a patient character, and at least one of the scenario tasks includes at least a dialogue task in which the patient character interacts with a task character in the virtual environment or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment, and at least a portion of the time, the virtual environment includes a representation of a coach character, and the method further includes: determining a psychological anxiety state parameter representative of the patient's psychological anxiety state by measuring one or more characteristics of an interaction between the patient character and the task character or the one or more objects in the virtual environment; monitoring performance of the scenario tasks in the virtual environment; and In response to one or both of the non-performance of the scenario task and the state of mental anxiety of the patient indicated by the mental anxiety state parameter, and assisting the patient in completing the scenario task in the virtual environment by providing VR feedback including one or both of verbal and visual prompts in the virtual environment using the coach character.
11. A program for causing one or more computers to carry out the method according to claim 10.
12. 1. A virtual reality (VR) therapy system for providing psychotherapy to a patient, comprising: A computer system; A patient receiving a patient diagnostic test, comprising: A head mounted display unit for people with hearing impairments, one or more sensors for detecting patient activities performed by the patient, the one or more sensors being coupled to the computer system; The computer system includes: The head mounted display unit is configured to display a virtual environment for a patient character, the virtual environment including a task character in a treatment scenario and a coach character, either temporarily or permanently, and the computer system further comprises: causing the head mounted display unit to prompt the patient to perform a predefined activity in relation to the task character; determining, using the one or more sensors, whether the patient performed the predefined activity within a first time period; The VR therapeutic system is configured to cause the head-mounted display unit to, in response to determining that the patient has not completed the predefined activity within the first period of time, cause the coach character to provide additional prompts to the patient to complete the predefined activity.
13. The computer system further comprises: determining whether the patient performs the predefined activity within a second time period after the additional prompt; in response to determining that the patient has performed the predefined activity, causing the head mounted display unit to progress the virtual environment to a different treatment scenario in the virtual environment or to a next level of the treatment scenario; 13. The VR therapy system of claim 12, configured to initiate an exit sequence to exit the patient from the virtual environment in response to determining that the patient has not performed the predefined activity within the second time period.
14. 14. The VR therapy system of claim 12 or 13, further comprising a hand controller system that tracks movements of the patient's hand, the hand controller system comprising a sensor of the one or more sensors, and the predefined activities include touching, holding, or moving a virtual object in the virtual environment.
15. The VR therapy system of any one of claims 12 to 14, wherein the one or more sensors include a microphone and the predefined activity includes talking to the task character.
16. 16. The VR therapy system of claim 12, wherein the computer system is further configured to obtain one or more input parameters related to the patient's behavior and adjust the first period based on the one or more input parameters.
17. 20. The VR therapy system of claim 16, wherein certain input parameters of the one or more input parameters depend on input received by the computer system from the one or more sensors.
18. 20. The VR therapy system of claim 17, wherein the one or more sensors include an eye-tracking device attached to the head mounted display unit, and the input parameter is dependent on a time spent by the patient looking at a predefined region of the virtual environment.
19. The VR therapy system of any one of claims 16 to 18, wherein certain of the one or more input parameters are retrieved by the computer system from a remote data store.
20. The VR therapeutic system of any one of claims 12 to 19, wherein the computer system is further configured to cause the head-mounted display unit to display an initial welcome virtual environment including the coach character but not the task character before displaying the therapeutic scenario.
21. 21. The VR therapeutic system of claim 12, wherein the computer system is configured to, in response to determining that the patient has performed the predefined activity, cause the head mounted display unit to progress the virtual environment to a next level of the therapeutic scenario.
22. 22. The VR therapy system of claim 12, further comprising a patient mobile device having a patient sensor input for sensing an anxiety level of the patient and an audio output, the patient mobile device being configured to process data from the patient sensor input to monitor the anxiety level of the patient when the patient is in the real world and away from the virtual environment and to provide an audio output in response to the sensed anxiety level, the audio output including instructions to the patient in a voice of the coach character.
23. 23. The VR therapy system of claim 22, wherein the patient mobile device comprises or consists of a wearable device for monitoring one or both of the patient's heart rate and galvanic skin response to sense the patient's anxiety level, and wherein the audio output is provided in response to detection of an increased anxiety level.
24. Further comprising a patient device and a remote server; the remote server is configured to receive, from the computer system, patient data collected while the patient is in the treatment scenario in the virtual environment, the patient data representing the patient's performance in the treatment scenario; The VR therapeutic system of any one of claims 12 to 23, wherein the patient device is configured to communicate with the remote server and output data to the patient in response to the patient's performance in the treatment scenario to facilitate the patient's performance in a real-world scenario corresponding to the treatment scenario.
25. the patient device is configured to collect further patient data dependent on the performance of the patient in the real-world scenario corresponding to the treatment scenario and to transmit the further patient data to the computer system via the remote server; 25. The VR therapy system of claim 24, wherein the remote server is further configured to adjust the therapy scenario according to the additional patient data such that the therapy scenario is adapted according to the performance of the patient in the real-world scenario.
26. The VR therapeutic system of any one of claims 1 to 8, further comprising a cortisol sensor for sensing a level of cortisol in the patient, the VR therapeutic system being configured to determine a degree of performance of the scenario task or predefined activity in response to a level of cortisol in the patient, or to determine a scenario task or predefined activity for presentation in the virtual environment in response to the level of cortisol in the patient.
27. The VR therapy system of any one of claims 1 to 8, wherein the VR therapy system is configured to collect patient data from the patient, the patient data characterizing the patient's performance in the scenario tasks or predefined activities, and the VR therapy system further comprises a machine learning subsystem or module configured to process the patient data to determine a therapy scenario or a level of a therapy scenario or several therapy scenarios or several levels of a therapy scenario to present to the patient in the virtual environment.
28. 1. A virtual reality (VR) therapy system for providing psychotherapy to a patient, comprising: The VR therapeutic system further comprises a VR system for providing a virtual environment, the VR system including at least a head mounted display unit having a sound generating capability and a VR input device. one or more computers coupled to the VR system or a part thereof, the one or more computers comprising: configured to present a set of treatment scenarios to the patient within the virtual environment using the VR system, each treatment scenario including at least one scenario task to be accomplished by the patient within the virtual environment by interacting with the VR system; the patient is represented in the virtual environment by a patient character, and at least one of the scenario tasks includes at least a dialogue task in which the patient character interacts with a task character in the virtual environment or a non-verbal activity task in which the patient character performs a non-verbal activity with respect to one or more objects in the virtual environment, and at least a portion of the time, the virtual environment includes a representation of a coach character, and the one or more computers further include: Implementing a patient state neural network having a neural network input to receive sensor data sensing one or more of: i) one or more characteristics of an interaction between the patient character and the task character or the one or more objects in the virtual environment; or ii) one or more of the patient's heart rate, galvanic skin response, and cortisol level in the real world; and processing the neural network input using the patient state neural network to determine a psychological anxiety state parameter representative of the patient's psychological anxiety state; monitoring performance of the scenario tasks in the virtual environment; In response to one or both of the non-performance of the scenario task and the state of mental anxiety of the patient indicated by the mental anxiety state parameter, The VR therapeutic system is configured to assist the patient in completing the scenario task in the virtual environment by: i) using the coach character to provide VR feedback including one or both of verbal and visual prompts in the virtual environment; or ii) modifying the treatment scenario to change the difficulty of the treatment scenario.
29. 29. The VR therapy system of claim 28, wherein the patient state neural network includes one or more initial input neural network layers configured to receive the neural network inputs and generate latent representations of the neural network inputs, at least one intermediate neural network layer having intermediate layer inputs and intermediate layer outputs coupled to the one or more input neural network layers, and one or more output layers having inputs coupled to the intermediate layer outputs to generate neural network outputs, the neural network outputs defining a score distribution over a discrete set of possible patient anxiety values, and the patient state neural network configured to generate the psychological anxiety state parameter from the score distribution.
Citation Information
Patent Citations
VR technology-based depression auxiliary treatment virtual scene system and realization method for same
CN107320836A
System for mental health clinical application
US20180011971A1
Content providing system, content providing method and program for content providing system
WO2018034113A1
Biofeedback for therapy in virtual and augmented reality
WO2019036051A1
