Apparatus, system and method for a real-time biocompatible stimulation environment

A real-time bioadaptive stimulation environment with AI/ML tailors psychedelic experiences through audio, visual, and olfactory components to address patient discomfort, enhancing therapy efficacy by adapting to individual responses.

JP2024544739AActive Publication Date: 2024-12-03COMPASS PATHFINDER LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024552100
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-23
Filing Date
2022-11-22
Publication Date
2024-12-03
Estimated Expiration
2042-11-22

AI Technical Summary

Technical Problem

Existing psychedelic therapies can be uncomfortable for patients due to intense initial experiences, necessitating a need for better patient preparation and experience customization.

Method used

A real-time bioadaptive stimulation environment using audio, visual, and olfactory components, integrated with biosensors and AI/ML, to tailor the psychedelic experience for individual patient responses, providing procedurally generated stimuli and continuous feedback for a positive therapeutic outcome.

Benefits of technology

Enhances patient comfort and therapeutic efficacy by adapting the experience in real-time to individual psychophysiological responses, ensuring a safe and effective psychedelic therapy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544739000001_ABST
    Figure 2024544739000001_ABST
Patent Text Reader

Abstract

An approach is provided for providing a real-time biocompatible stimulation environment with audio, visual, and olfactory components to enhance psychedelic therapy. A virtual biocompatible environment may be provided for experience by a user. Sensor data associated with the user may be received. The sensor data may be analyzed using at least one machine learning model to determine a change in user state. Modifications to be made to the virtual biocompatible environment may be determined based at least in part on the analyzed sensor data. The modified virtual biocompatible environment may be provided to the user.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This PCT application claims priority to U.S. Provisional Patent Application No. 63 / 282,635, entitled “APPARATUSES, SYSTEMS, AND METHODS FOR A REAL TIME BIOADAPTIVE STIMULUS ENVIRONMENT,” filed November 23, 2021, which is incorporated by reference in its entirety for all purposes. [Background technology]

[0002] Since the advent of modern medicine, many treatments and drug schedules have been developed. However, while many such treatments can be effective, the initiation of some treatments can be unpleasant for some patients. For example, in the case of psychedelic-mediated treatments, many patients have never had a psychedelic experience and may find their initial experience to be rather intense.

[0003]

[0003] It may be beneficial to prepare a patient for an upcoming treatment prior to administration of medication. Collected patient data may be analyzed to help assess how a patient may be feeling at a given time point. Furthermore, collected data may be processed using deep learning and machine learning techniques to help tailor the patient's experience both during and after medication administration. Summary of the Invention

[0004]

[0004] Various embodiments according to the present disclosure are described with reference to the drawings. [Brief description of the drawings]

[0005] [Figure 1A] 1 is an exemplary scene diagram that may be utilized in accordance with one or more embodiments. [Figure 1B]1 is an exemplary scene diagram that may be utilized in accordance with one or more embodiments. [Figure 2A]

[0006] FIG. 1 illustrates example sensor data that may be utilized in accordance with one or more embodiments. [Figure 2B] FIG. 1 illustrates example sensor data that may be utilized in accordance with one or more embodiments. [Diagram 3]

[0007] FIG. 1 illustrates an example system that may be utilized to implement one or more aspects of various embodiments. [Figure 4]

[0008] FIG. 1 illustrates an example methodology that may be utilized to implement one or more aspects of various embodiments. [Diagram 5]

[0009] FIG. 1 illustrates an example of an environment that can be utilized to implement one or more aspects of various embodiments. [Figure 6]

[0010] FIG. 1 illustrates an example of an environment for implementing one or more aspects of various embodiments. [Figure 7]

[0011] 1 is an exemplary block diagram of an electronic device that may be utilized to implement one or more aspects of the various embodiments. [Figure 8]

[0012] FIG. 1 illustrates components of another exemplary environment in which aspects of various embodiments may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006]

[0013] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the embodiments being described.

[0007]

[0014] A real-time bioadaptive stimulation environment with audio, visual, and olfactory components for enhancing psychedelic therapy is provided. A system associated with the environment may be configured to provide a real-time bioadaptive sensory environment that uses low-latency and time-synchronized objective psychophysiological measurements to adapt the psychedelic experience to the temporal evolution of the patient's subjective state in a medication session. A biosensor suite associated with the environment may record measurements and adapt audio, visual, and olfactory stimuli to the patient's subjective state as it evolves to promote a positive psychedelic experience that may lead to a desired therapeutic outcome. According to an exemplary embodiment, the stimuli may be adapted using artificial intelligence and machine learning techniques.

[0008]

[0015] The system may also provide the patient with real-time procedurally generated direct visual, olfactory, and auditory stimuli to help guide each patient through a unique psychedelic experience. For example, the psychedelic experience may be adapted and / or controlled to produce a desired therapeutic outcome. As a further example, if the machine learning model determines that the patient is uncomfortable and would benefit from listening to a calming sound, the system may play such a calming sound for the patient to listen to. In at least this example, the system may operate in real-time, and the patient's response to the calming sound may be measured and interpreted by the system, which allows the system to further modify the sound output. The sound output may be modified in volume and may also be modified to play different types of sounds, such as melodies, single tones, or combinations of tones.

[0009]

[0016] 1A and 1B show exemplary scene diagrams 100, 110 that may be utilized according to one or more embodiments. According to one exemplary embodiment, a virtual reality (VR)-enabled setting for preparation, dosing, and integration sessions may be utilized to help enhance the psychedelic experience. For example, the VR setting may include either or both of an environmental setting, such as scene diagram 100, and an avatar guide 120 shown in scene diagram 110. While this example refers to the use of VR, augmented reality or enhanced reality may also be utilized according to various embodiments. In an environmental setting, according to one exemplary embodiment, the patient may be able to experience a three-dimensional world and environment (e.g., a forest or a beach), while in an avatar guide, the patient may interact with a persona that guides the patient through the experience. The system may utilize an environmental setting, an avatar guide setting, and / or a combination of both settings. In one exemplary embodiment, enhancements for both preparation for the experience and / or post-experience review may be utilized. In this example, the system may record which stimuli were presented and the time at which those stimuli were presented, which allows for an experience "playback." Additionally, the system may record and store audio and / or visual output associated with the patient, so that the therapist or patient can later view the patient's experience from a different perspective, such as a perspective view of the patient. In some exemplary embodiments, the perspective view and the stimulation schedule for the session may be superimposed or overlaid, so that upon review of the session, the reviewer can observe the patient while also reading the stimuli.

[0010]

[0017] In at least one exemplary embodiment, the system may include a patient virtual profile. The patient virtual profile may be created with a priori settings, which may then be modified based on the patient experience. The settings of the patient virtual profile may be initially populated when the user first registers. For example, the system may inquire about various personal details at registration. In another example, a therapist, administrator, or other user may determine the settings. The patient may then interact with the system to prepare for a session, such as a medication session, by experiencing a preparatory experience, such as a simulated psychedelic experience, involving immersive video and audio stimuli, such as those provided by a VR headset. According to one exemplary embodiment, psychophysiological patient responses may be monitored and recorded. In this example, the recorded responses may be utilized to calibrate the medication session and build a posteriori profile of optimal experience settings. As a non-limiting example, the experience may adjust the person's environment, the music the person listens to, or the story the patient is exposed to in the VR environment. These changes modify the set and configuration of the psychedelic experience and can therefore be used to guide the patient towards a particular emotional response, such as calmness, excitement, or curiosity. Because each person responds differently to different stimuli, calibration can be used to learn how an individual responds to different sensory inputs. In alternative embodiments, the responses can be used to calibrate the system for any type of treatment and / or session.

[0011]

[0018] 2A and 2B show example sensor data 200, 210 that may be utilized according to one or more embodiments. According to one example embodiment, the system may be configured to determine photoplethysmogram (PPG) respiration, PPG heart rate, electrocardiogram (ECG) data, and electroencephalogram (EEG) data, among other measurements. The system may be further configured to determine brain electrical activity, such as by using an EEG sensor. The EEG sensor may include, but is not limited to, a four-channel headband that measures the left and right temporoparietal lobes and the left and right prefrontal lobes.

[0012]

[0019] Also, during administration of medication, the patient may be monitored through various hardware systems and sensors, including but not limited to, cameras (including but not limited to high definition calibrated camera arrays capable of monitoring heart rate / pulse, respiration, temperature, flushing response, facial expressions, and / or pupillary response), microphones (including but not limited to beamforming microphone arrays for capturing spatially localized audio), electroencephalography, wearables (including but not limited to wrist-based EMG wearables and ECG chest straps), and other suitable hardware components.

[0013]

[0020] According to an exemplary embodiment, the system may include a computer in the same vicinity as the sensors configured to locally record and process the time-synchronous signals. In this example, the recorded signals and / or results may be uploaded to a cloud infrastructure for later post-processing. Machine learning models, such as at-the-edge machine learning models, may be trained to simultaneously process various information to tailor the psychedelic experience through adjustments to the visual, audio, and olfactory stimuli environment presented to the patient. In an exemplary embodiment, the system may modify the immersive visual environment experienced by the patient. Using the analyzed sensor data, the system may determine, using machine learning, which environmental modifications should be made such that those modifications are made automatically in real-time or near real-time. For example, if a given data point in the sensor data falls below a determined threshold or score, the system may determine that corrective action is required. Depending on the particular data point, the system may determine which one or more stimuli should be adjusted. For example, a visual representation of a beach may be provided for presentation instead of a library or a river. Additionally, the accompanying audio may also be modified. For example, classical music may be changed to jazz, or stimulating music may be changed to soothing music. Olfactory stimuli may also be changed. For example, the refreshing smell of a river may be provided compared to the musty smell of a library. Additionally, audio pitch, audio volume, scene brightness, avatar type, scent type, or scent intensity may be adjusted manually or automatically, among other options. According to an exemplary embodiment, continuous psychophysiological data may be taken from the patient to further adjust the adapted stimulation environment, as well as to update machine learning models that may be guiding the session.

[0014]

[0021] After dosing, the patient may be able to recreate part of the patient's psychedelic experience through replaying the same sensory stimuli the patient experienced while on medication, under supervision, as appropriate, to aid integration. Group integration experiences may be held virtually, as appropriate and depending on the treatment profile. For example, multiple patients may interact with multiple biosensor suites, or multiple patients may interact with the same biosensor suite.

[0015]

[0022] 3 illustrates an exemplary system 300 that may be utilized to implement one or more aspects of various embodiments. According to one exemplary embodiment, the system 300 may assist the entire treatment process. For example, the system 300 may be able to assist the patient during the preparation, dosing, and integration stages. Moreover, the system may enhance the safety, efficacy, and accessibility profile of the treatment, such as by better regulating the experience and / or deploying the experience on a larger scale.

[0016]

[0023] According to an exemplary embodiment, the system 300 may include a sensor suite 302 with associated sensor data. The sensor suite 302 may provide real-time monitoring of the patient 308 and / or therapist 310 in the therapy environment 306. In an exemplary embodiment, the therapist 310 may have a subset of sensors that are either visible or invisible to the patient 308. For example, electromyography (EMG) may be used to help analyze nerve-to-muscle signal transmission. Sensors specific to the therapist 310 may be configured to not affect the patient experience. In at least one exemplary embodiment, a camera suite 312 may also be utilized to help collect visual data about the patient 308 and / or therapist 310. The sensor suite devices may wirelessly sync to a compact server, such as a single board computer. According to an exemplary embodiment, one or more devices of the sensor suite 302 may be synced via a wired or wireless connection. Machine learning models may be employed to facilitate the adjustment of visual, audio, and / or olfactory stimuli using one or more stimulation devices 304. The adjustment of the various stimuli may be facilitated, at least in part, using virtual reality devices, augmented reality devices, enhanced reality devices, or any other such presentation devices. Additionally, the stimuli may be adjusted using a combination of devices, such as a speaker with an audio output, or a device that may emit various scents. During or after the monitoring, the data may be relayed to a cloud-based environment 314 for further analysis. The analysis may be performed, for example, by using machine learning. Over the course of or after the monitoring, the therapist may be able to review the biofeedback process on an electronic device, including, but not limited to, a smartphone, personal computer, or tablet. The system may also maintain and / or facilitate a cloud-based infrastructure that allows various biocompatible software to be distributed and accessed from a centralized repository.Such a system may allow for continuous virtual software updates for a given hardware profile.

[0017]

[0024] The system may be used for several applications. For example, the system may provide a suite of biosensors at the edge that objectively measure patient psychophysiological biomarkers, including, but not limited to, electroencephalography, pulse, electrocardiogram, facial expressions and flushing response, pupillary response, muscle tone, and / or electrodermal response, for both immediate processing and long-term cloud-enabled storage. Data correlating to individual metrics (e.g., pulse, facial expressions, flushing response, etc.) may be stored locally and / or in cloud storage. Data may be stored in raw or processed format. In various embodiments, analysis of sensor information to identify biomarkers and provide continuous feedback may be performed in the cloud or at the edge depending on the complexity of the model.

[0018]

[0025] The sensor suite 302 may include, but is not limited to, electroencephalography (EEG), electrocardiography (ECG), photoplethysmography, pulse oximetry, electromyography (EMG), spatial audio recording, and / or a camera array. The camera array may include high resolution color (RGB), thermal, and / or depth sensors with sufficient frame rates to measure in real time pulse, respiration, temperature changes, facial flushing response, facial expressions, pupillometry, eye movements, and general physical movements that may indicate physical agitation. Sensors may also be embedded in hardware, such as motion sensors in a VR headset.

[0019]

[0026] According to an exemplary embodiment, the sensor suite 302 may enable various flows of communication. According to non-limiting examples, the communication may include person-to-person interaction, information flow of data from person to device, or stimulation from device to person. In an exemplary embodiment, the sensor suite 302 may be configured to receive data from the patient 308. Stimulation hardware (e.g., visual, auditory, and olfactory measurements) may be configured to deliver immersive stimulation 304 to the patient 308. The patient 308 and therapist 310 may be in person-to-person communication. The camera suite 312 may include one or more microphone arrays and may be configured to receive data from the patient 308 and / or therapist 310. The patient 308, therapist 310, camera suite 312, and other sensor devices may be in bidirectional communication with the cloud-based environment 314. In various embodiments, any of the components may be configured to communicate with any of the other components and / or parties.

[0020]

[0027] The system may include a login portal configured for anonymity during a clinical trial setting, according to one exemplary embodiment. Additionally, the system may be configured for automated cross-platform or cross-operating system digital biomarker and sensor data collection that may be incorporated into a single cloud-native database. The system may also provide a synchronized content database that may be updated remotely for new studies and patients.

[0021]

[0028] According to an exemplary embodiment, the front-end application may be used across all operating systems, mobile devices, and personal computers. The front-end application may allow easier validation of the application for regulatory purposes as one code base. The application may allow for custom biomarker display to the care team and patient, as well as custom alerts based on collected data. The application may allow for "full circle" machine learning, as the system may be able to collect data, upload data, analyze data, identify triggers, and send information to the care team.

[0022]

[0029] FIG. 4 illustrates an example method 400 that may be utilized to implement one or more aspects of various embodiments. It should be understood that for any process herein, there may be additional, fewer, or alternative steps performed in a similar or alternative order or in parallel within the scope of various embodiments, unless otherwise stated. According to an example embodiment, a virtual biocompatible environment may be provided 410 for experience by a user. The virtual biocompatible environment may be provided, in at least some embodiments, on a virtual reality device, an augmented reality device, an enhanced reality device, or any other such presentation device or system. Sensor data associated with the user may be received 420, such as through the presentation device or system(s). The sensor data may be analyzed 430 using at least one machine learning model to determine one or more changes in a user state. For example, it may be determined that at least a subset of the sensor data falls below a determined threshold level. Based at least in part on the analyzed sensor data, one or more modifications to be made to the virtual biocompatible environment may be determined 440. The modified virtual biocompatible environment may be provided 450 on a presentation device(s) or system.

[0023]

[0030] FIG. 5 illustrates an example of an environment 500 that may be utilized to implement one or more aspects of various embodiments. According to an exemplary embodiment, the environment may be a computational layer. A sensor suite may provide sensor data 502 and may be in communication with a computer or processor 504. The computer 504 may be in communication with a user management node 506 so that a therapist, patient, or other party may be authenticated. According to an exemplary embodiment, a patient may subscribe to the system's digital infrastructure when the patient is prescribed treatment. Upon registration, a unique identifier may be assigned to the patient. The unique identifier may be used for future tracking and for potential integration with other components of the system or with tertiary systems such as companion applications. The unique identifier may be used as or with metadata that may be tagged onto data that correlates to that patient. For example, a video file collected by a camera array or an audio file collected by a microphone may be correlated with the unique identifier and metadata. Prior to dosing, the patient may be able to review settings, as may be appropriate, through the system infrastructure (e.g., a mobile application, web portal, or suitable alternative) to fully prepare for the dosing experience.

[0024]

[0031] The computer 504 may also be in communication with a cloud-based environment 508 via a secure application programming interface (API) gateway 510. The cloud-based environment 508 may include or be in communication with an authentication database 512, which is in communication with a user management node 506 and configured to help manage users. The cloud-based environment 508 may include a session data upload 514, which may extract, transform, and load data into a structured data store 516. The structured data store 516 may be in communication with a deep learning or machine learning optimization node 518. The deep learning or machine learning optimization node 518 may be configured to analyze both a priori patient data and per-patient (later model) data to create an optimal experience that is specifically tailored for the patient. In an exemplary embodiment, the deep learning or machine learning model optimization node may receive information regarding the global population as well as information including the patient's specific or particular preferences.

[0025]

[0032] As a non-limiting example, if ocean sounds are universally soothing and the patient prefers jazz music over classical music for relaxation, the deep learning or machine learning model optimization node may create or generate a custom experience that combines both ocean sounds and the jazz genre. Additionally, the deep learning or machine learning model optimization node may also modulate the experience, such as by selecting a particular scene or music. In such an embodiment, each particular image or audio file may have varying intensities of various qualities. For example, a first beach scene may be very soothing, while a second beach scene may be somewhat soothing. Thus, in such an example, the deep learning or machine learning model optimization node may discriminate between particular instances of each class of stimuli and present each particular instance based on global population information and / or patient-specific information. The patient model store 520 may then receive the optimized data from the deep learning or machine learning model optimization node. In an exemplary embodiment, the patient model store may be in communication with a "model as a service" (MaaS) 522 to provide real-time on-demand models for each patient. To provide the model for each patient, the MaaS may be in communication with a computer 504 via a secure API gateway 510.

[0026]

[0033] The system associated with the environment may utilize various deep learning, general machine learning and / or statistical models, which are evaluated for optimal performance. For example, the system may employ any number or combination of the following models: perceptron, feedforward, radial basis network, deep feedforward, recurrent neural network, long / short-term memory, gated recurrent unit, autoencoder, variational autoencoder, sparse autoencoder, denoising autoencoder, Marcus chain, Hopfield network, Boltzmann machine, restricted Boltzmann machine, deep belief network, deep convolutional network, deep network, deep convolutional inverse graphic network, generative adversarial network, liquid state machine, extreme learning machine, echo network machine, Kohonen network, deep residual network, support vector machine, and neural Turing machine.

[0027]

[0034] The system may include a layer of computation at the edge. The sensor suite may be synchronized to the computer or computing device during a dosing session. The computer may be preloaded with a global machine learning model that may be overwritten or updated from a cloud-based network setting. The global machine learning model may not take into account patient preferences and may be used if prior information from the patient is not available or required. The patient-specific machine learning model may incorporate information from the patient (either in real-time or during training) via the deep learning or machine learning model optimization node described above. The model may run in real-time and provide feedback to the device for the patient and / or therapist. A user profile may be downloaded from the cloud along with additional model parameters as needed. User biomarkers and sensor data for either or both the patient and therapist may be uploaded to the cloud-based network for further updates and evaluation. In some exemplary embodiments, the system may develop a continuous feedback system that allows an administrator, user, or medical professional to improve the baseline psychedelic experience and develop a unique patient-specific profile that can be used for subsequent dosing sessions. For example, a patient-specific profile may be stored in a cloud-based environment and may include default values ​​or baselines to be utilized by the system for the patient's next session. In this example, by utilizing past data, the system may reduce the amount of time required to effectively match a patient to a medication. Additionally, the system may also predict and adjust future baselines or default values ​​based on the current profile and past data.

[0028]

[0035] The system may include and / or follow a patient treatment timeline. The preparation step may include providing a pre-medication "psychedelic experience" to prepare the patient for the medication session. The preparation step may also include measuring the patient's response to the pre-medication to create a patient-specific digital experience. Additionally, during this pre-medication step, the system may be configured to identify and / or flag early warning signs and attempt to mitigate potential issues.

[0029]

[0036] The dosing step may include providing a customized digital experience by adjusting visual, olfactory, and auditory stimuli. In this exemplary embodiment, a real-time biofeedback loop may use the model at the edge to adjust the experience to increase patient safety and improve treatment delivery. Additionally, in the dosing step, the system may record sensor information for the integration session and for model updates. According to one exemplary embodiment, the dosing session may involve administering some form of psilocybin to the patient.

[0030]

[0037] The integration step may include activating memory of the experience by replaying sensory feedback experienced during the administration session, such as audio, olfactory, and visual. The integration step may enable in-group integration through the VR infrastructure. Furthermore, according to an exemplary embodiment, multiple self-guided integration sessions may be possible. In an embodiment, the system may update both the global model and the patient-specific model for future dosing sessions.

[0031]

[0038] During dosing, the environment can be used and adjusted to modulate both the intensity and emotional valence of the experience based directly on patient feedback as well as broader population-level data. In a further embodiment, the system can improve the integration session by giving the patient a way to replay any part of the patient's experience for further reflection. Such environments can be used throughout the patient journey to further enhance the treatment model.

[0032]

[0039] As described, different approaches may be implemented in various environments according to the described embodiments. For example, FIG. 6 illustrates an example of an environment 600 for implementing one or more aspects of various embodiments. As will be appreciated, although a web-based environment is used for purposes of explanation, different environments may be used to implement various embodiments, as appropriate. The system includes electronic client devices 602, 608, which may include any suitable device operable to send and receive requests, messages or information over a suitable network 604 and to communicate information to a user of the device. Examples of such client devices include personal computers, virtual reality device(s), augmented reality device(s), enhanced reality device(s), cell phones, handheld messaging devices, laptop computers, set-top boxes, personal digital assistants, e-book readers, and the like. The network may include any suitable network, including an intranet, the Internet, a cellular network, a local area network, or any other such network, or combinations thereof. The components used for such a system may depend at least in part on the type of network and / or environment selected. Protocols and components for communicating over such networks are well known and will not be described in detail herein. Communication over the network may be enabled via wired or wireless connections and combinations thereof. In this example, the network includes the Internet as the environment includes one or more servers 606 for receiving requests and serving content in response thereto, although, as will be apparent to one skilled in the art, in other networks, alternative devices serving similar purposes may be used.

[0033]

[0040] The exemplary environment includes at least one application server 610 and a data store 612. It should be understood that there may be several application servers, layers or other elements, processes or components, which may be coupled or otherwise configured, that may interact to perform a task, such as retrieving data from an appropriate data store. The term "data store" as used herein refers to any device or combination of devices capable of storing, accessing and retrieving data, which may include any combination of data servers, databases, data storage devices and data storage media, and any number of such devices, in any standard, distributed or clustered environment. The application server 610 may include any suitable hardware and software for integrating with the data store 612 as needed to execute aspects of one or more applications for client devices, and for handling much of the data access and business logic for the applications. The application server may cooperate with the data store to service access control and generate content, such as text, graphics, audio and / or video to be transferred to the user, which may be served to the user by one or more servers 606, including a web server, in this example, in the form of HTML, XML or another suitable structured language. Handling of all requests and responses, as well as delivery of content between the client devices 602, 608 and the application server 610 may be handled by the web server of the server 606. It should be understood that the web server and application server are not required and are merely exemplary components, as the structured code described herein may be executed on any suitable device or host machine, as described elsewhere herein.

[0034]

[0041] The data store 612 may include a number of separate data tables, databases, or other data storage mechanisms and media for storing data related to a particular aspect. For example, the illustrated data store includes mechanisms for storing production data 614 and user information 618, which may be used to serve content for the manufacturing side. The data store is also shown to include mechanisms for storing log or application session data 616. It should be understood that there may be many other aspects that may need to be stored in the data store, such as page image information and access rights information, which may be stored in any of the mechanisms listed above or in additional mechanisms in the data store 612, as appropriate. The data store 612, through logic associated therewith, is operable to receive instructions from the application server 610 and retrieve, update, or otherwise process data in response thereto. In one example, a user may submit a request to transcribe, tag, and / or label a media file. In this case, the data store may access the user information to verify the user's identity and provide a transcript including the tags and / or labels along with analytics associated with the media file. The information may then be returned to the user, such as in a results listing on a web page that the user can view via a browser on the user device 602, 608. Information about a particular feature of interest may be viewed in a dedicated page or window of the browser.

[0035]

[0042] Each server will typically include an operating system that provides executable program instructions for the general administration and operation of the server, and will typically include a computer-readable medium that stores instructions that, when executed by the server's processor, enable the server to perform its intended functions. Suitable implementations for server operating systems and general functions are known or commercially available, and will be readily implemented by those of skill in the art, especially in light of the disclosure herein.

[0036]

[0043] The environment in one embodiment is a distributed computing environment utilizing several computer systems and components interconnected via communication links using one or more computer networks or direct connections. However, those skilled in the art will appreciate that such a system may operate equally well in a system having fewer or a greater number of components than are shown in Figure 6. Thus, the illustration of system 600 in Figure 6 should be taken as illustrative in nature and not limiting on the scope of the present disclosure.

[0037]

[0044] 7 illustrates an example block diagram of an electronic device that may be utilized to implement one or more aspects of various embodiments. An instance of an electronic device 700 may include one or more servers and one or more client devices. In general, the electronic device may include a processor / CPU 702, a memory 704, a power supply 706, and input / output (I / O) components / devices 710 that may be operable to provide, for example, a graphical or textual user interface, such as a microphone, speaker, display, touch screen, keyboard, mouse, keypad, microscope, GPS components, camera, heart rate sensor, light sensor, accelerometer, target biometric sensor, neck wearable to detect brain activity, etc.

[0038]

[0045] A user may provide input via a touchscreen of the electronic device 700. The touchscreen may determine whether a user is providing input, for example, by determining whether the user is touching the touchscreen with a part of the user's body, such as the user's finger. The electronic device 700 may also include a communication bus 712 that connects to the above-mentioned elements of the electronic device 700. The network interface 708 may include a receiver and transmitter (or transceiver) and one or more antennas for wireless communication.

[0039]

[0046] The processor 702 may include any type of processing device, such as one or more of a central processing unit (CPU) and a graphics processing unit (GPU). Also, for example, the processor may utilize central processing logic, or other logic, for performing one or more functions or actions or for causing one or more functions or actions from one or more other components, and may include hardware, firmware, software, or a combination thereof. Also, based on a desired application or need, the central processing logic, or other logic may include, for example, a software-controlled microprocessor, discrete logic, such as an application specific integrated circuit (ASIC), a programmable / programmed logic device, a memory device containing instructions, or the like, or combinational logic embodied in hardware. Additionally, logic may also be embodied entirely as software.

[0040]

[0047] Memory 704, which may include random access memory (RAM) 714 and read only memory (ROM) 716, may be enabled by any type of memory device, for example, one or more of a primary (directly accessible by the CPU) or secondary (indirectly accessible by the CPU) storage device (e.g., flash memory, magnetic disk, optical disk, etc.). The RAM may include an operating system 718, data storage 720, which may include one or more databases, and programs and / or applications 722, which may include, for example, software aspects of programs 724. The ROM 716 may also include a basic input / output system (BIOS) 726 of the electronic device 700.

[0041]

[0048] The software aspects of the programs 722 are intended to broadly include or represent all programming, applications, algorithms, models, software, and other tools necessary to implement or facilitate methods and systems according to embodiments of the present invention, the elements of which may reside on a single computer or be distributed among multiple computers, servers, devices, or entities.

[0042]

[0049] The power supply 706 may include one or more power components and may help facilitate the supply and management of power to the electronic device 700.

[0043]

[0050] Input / output components including input / output (I / O) interface 710 may include any interface for facilitating communication between, for example, components of electronic device 700, components of external devices, and end users. For example, such components may include a network card, which may be an integration of a receiver, a transmitter, a transceiver, and one or more input / output interfaces. The network card may facilitate wired or wireless communication, for example, with other devices of a network. In the case of wireless communication, an antenna may facilitate such communication. Also, some of the input / output interfaces 710 and the bus 712 may facilitate communication between components of electronic device 700 and, in one example, may facilitate processing performed by processor 702.

[0044]

[0051] If the electronic device 700 is a server, the server may include a computing device that may be capable of sending or receiving signals, e.g., a wired or wireless network, or may be capable of processing or storing signals, e.g., in memory, as physical memory states. The server may be an application server that includes configurations for providing one or more applications to another device over a network. The application server may also host a website that may provide, for example, a user interface for administration of the exemplary embodiment.

[0045]

[0052] FIG. 8 illustrates an example environment 800 in which aspects of various embodiments may be implemented. In this example, a user may utilize one or more client devices 802 to submit requests to a multitenant resource provider environment 806 over at least one network 804. The client devices may include any suitable electronic device operable to send and receive requests, messages, or other such information over an appropriate network and to communicate information to a user of the device. Examples of such client devices include personal computers, virtual reality device(s), augmented reality device(s), enhanced reality device(s), tablet computers, smartphones, notebook computers, and the like. The at least one network 804 may include any suitable network, including an intranet, the Internet, a cellular network, a local area network (LAN), or any other such network, or combination, and communication over the network may be enabled via wired and / or wireless connections. The resource provider environment 806 may include any suitable components for receiving requests and returning information or performing actions in response to those requests. As an example, the provider environment may include web servers and / or application servers for receiving and processing requests and then returning data, web pages, video, audio, or other such content or information in response to the request.

[0046]

[0053] In various embodiments, the provider environment may include various types of resources that may be utilized by multiple users for various different purposes. As used herein, computing resources and other electronic resources utilized in a network environment may be referred to as "network resources." These may include, for example, servers, databases, load balancers, routers, and the like that may perform tasks, such as receiving, transmitting, and / or processing data and / or executable instructions. In at least some embodiments, all or a portion of a given resource or set of resources may be allocated to a particular user or for a particular task, at least for a determined period of time. The sharing of these multi-tenant resources from the provider environment is often referred to as resource sharing, web services, or "cloud computing," among other terms, as well as depending on the particular environment and / or implementation. In this example, the provider environment includes multiple resources 814 of one or more types. These types may include, for example, application servers operable to process instructions provided by users, or database servers operable to process data stored in one or more data stores 816 in response to user requests. As is known for such purposes, a user may also reserve at least a portion of the data storage in a given data store. Methods for enabling users to reserve various resources and resource instances are well known in the art, and thus a detailed description of the entire process, and an explanation of all possible components, will not be described in detail herein.

[0047]

[0054] In at least some embodiments, a user desiring to utilize a portion of the resource 814 can submit a received request to an interface layer 808 of the provider environment 806. The interface layer can include an application programming interface (API) or other exposed interface that allows a user to submit a request to the provider environment. The interface layer 808 in this example can also include other components, such as at least one web server, a routing component, a load balancer, etc. When a request to provision a resource is received in the interface layer 808, information about the request can be directed to a service manager 810, or other such system, service, or component, configured to manage user accounts and information, resource provisioning and usage, and other such aspects. The service manager 810 that receives the request can perform tasks such as authenticating the identity of the user who submitted the request and determining whether the user has an existing account with the resource provider, if account data can be stored in at least one data store 812 in the provider environment. The user can provide any of a variety of types of credentials to the provider to authenticate the user's identity. These credentials may include, for example, a username and password pair, biometric data, a digital signature, a QR-based certificate, or other such information.

[0048]

[0055] The provider can verify this information against information stored about the user. If the user has an account with the proper permissions, status, etc., the resource manager can determine whether there are sufficient resources available to suit the user's request, and if there are sufficient resources, can provision those resources for use by the user for the amount specified by the request, or possibly grant access to a corresponding portion of those resources. This amount can include, for example, the ability to process a single request or perform a single task, a specified time period, or a cycling / renewable period, among other values. If the user does not have a valid account with the provider, or the user account does not allow access to the type of resource specified in the request, or another such reason prevents the user from gaining access to such resources, a communication can be sent to the user to enable the user to create or modify an account or change the resource specified in the request, among other options. In at least some exemplary embodiments, a user can be authenticated to access an entire fleet of services offered within the service provider environment. In other exemplary embodiments, the user's access can be limited to specific services within the service provider environment using one or more access policies tied to the user's credential(s).

[0049]

[0056] Once a user has been authenticated, an account verified, and resources allocated, the user may utilize the allocated resource(s) for a specified capacity, amount of data transfer, time period, or other such value. In at least some embodiments, the user may provide a session token or other such proof with subsequent requests to allow those requests to be processed on that user session. The user may receive a resource identifier, specific address, or other such information that may enable the client device 802 to communicate with the allocated resource without having to communicate with the service manager 810, at least until such time that a relevant aspect of the user account changes, the user is no longer granted access to the resource, or another such aspect changes.

[0050]

[0057] Also, the service manager 810 in this example (or another such system or service) can function as a virtual layer of hardware and software components that handles control functions in addition to management actions, such as may include provisioning, scaling, replication, etc. The resource manager can utilize dedicated APIs in the interface layer 808, where each API can be provided to receive a request for at least one specific action to be performed with respect to the data environment, such as provisioning, scaling, cloning, or quiescing an instance. Upon receiving a request to one of the APIs, the web services portion of the interface layer can parse or otherwise analyze the request to act on the call or determine the steps or actions required to process the call. For example, a web services call can be received that includes a request to create a data repository.

[0051]

[0058] The interface layer 808 in at least one embodiment includes a scalable set of user-facing servers that can offer various APIs and return appropriate responses based on the API specifications. The interface layer, in one embodiment, can also include at least one API services layer, consisting of stateless, replicated servers that handle external-facing user APIs. The interface layer can be responsible for web services front-end features such as authenticating users based on credentials, authorizing users, throttling user requests to the API server, validating user input, and marshalling or unmarshalling requests and responses. The API layer can also be responsible for reading and writing database configuration data to / from an administration data store in response to API calls. In many embodiments, the web services layer and / or API services layer will be the only externally visible components or the only components that are visible to and accessible by users of the control services. The servers of the web services layer are stateless and can be horizontally scaled as known in the art. The API servers, as well as the persistent data stores, may be spread across multiple data centers across a region, for example, so that they are resilient to single data center failures.

[0052]

[0059] Various embodiments may also be implemented in a wide variety of operating environments that may, in some cases, include one or more user computers or computing devices that may be used to operate any of a number of applications. User or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as any of a number of cellular, wireless and handheld devices capable of running mobile software and supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming systems and other devices capable of communicating over a network.

[0053]

[0060] Most embodiments utilize at least one network, as will be familiar to those skilled in the art, for supporting communications using any of a variety of commercially available protocols, such as TCP / IP, FTP, UPnP, NFS, and CIFS. The network may be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. In embodiments utilizing a web server, the web server may run any of a variety of server or mid-tier applications, including an HTTP server, an FTP server, a CGI server, a data server, a Java server, and a business application server. The server(s) may also be capable of executing programs or scripts in response requests from user devices, such as by executing one or more web applications, which may be implemented as one or more scripts or programs written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, or Tcl, and combinations thereof. The server(s) may also include database servers, including, but not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, and IBM®.

[0054]

[0061] The environment may include various data stores and other memory and storage media, as described above. These may be present in various locations, such as across a network, on storage media local to (and / or present in) one or more of the computers, or remote from any or all of the computers. In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, necessary files to perform functions ascribed to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., a mouse, keyboard, controller, touch-sensitive display element or keypad), and at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices, such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash cards, etc. Such devices may also include a computer-readable storage medium reader and a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device), and a working memory as described above. A computer-readable storage medium reader may be connected to or configured to accept computer-readable storage media representing remote, local, fixed and / or removable storage devices, as well as storage media for temporarily and / or more permanently containing, storing, transmitting and retrieving computer-readable information.

[0055]

[0062] The systems and various devices will also include several software applications, modules, services or other elements located in at least one working memory device, including an operating system and application programs such as client applications or web browsers. It should be appreciated that alternative embodiments may have numerous variations from the embodiments described above. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, software (including portable software such as applets), or both. Additionally, connections to other computing devices, such as network input / output devices, may be employed. Storage media and other non-transitory computer readable media for containing code or portions of code may include any suitable media known or used in the art, such as volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data, including, but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other media that may be used to store the desired information and that may be accessed by the system devices. Based on the disclosure and teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various embodiments.

[0056]

[0063] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, however it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

Claims

1. providing a virtual biocompatible environment for experience by a user from a presentation device; receiving sensor data associated with the user from the presentation device; analyzing the sensor data using a machine learning model to determine one or more changes in a user state; determining one or more modifications to be made to the virtual biocompatible environment based at least in part on the analyzed sensor data; and providing a modified virtual biocompatible environment on said presentation device; and A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the virtual biocompatible environment includes at least one of audio stimuli, visual stimuli, and olfactory stimuli.

3. The computer-implemented method of claim 1 , wherein the virtual biocompatible environment is provided at least in part using virtual reality, augmented reality, or augmented reality.

4. The computer-implemented method of claim 2 , wherein the visual stimuli include at least one of a scene image and an avatar guide.

5. The computer-implemented method of claim 1 , wherein the virtual biocompatible environment is automatically modified in real-time or near real-time.

6. storing the modified virtual biocompatible environment in a user profile specific to the user; The computer-implemented method of claim 1 , further comprising:

7. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to: providing a virtual biocompatible environment for experience by a user from a presentation system; receiving sensor data associated with the user from the presentation system; analyzing the sensor data using a machine learning model to determine one or more changes in a user state; providing a modified virtual biocompatible environment on the presentation system based at least in part on the analyzed sensor data; and A non-transitory computer-readable medium for causing

8. The instructions, when executed by the at least one processor, cause the at least one processor to: determining, based at least in part on the analyzed sensor data, that at least a subset of the sensor data is below a threshold level; providing the modified virtual biocompatible environment based at least in part on the subset of the sensor data being below the threshold level; and The non-transitory computer-readable medium of claim 7 , further comprising:

9. 8. The non-transitory computer-readable medium of claim 7, wherein the virtual biocompatible environment includes at least one of audio stimuli, visual stimuli, and olfactory stimuli.

10. 8. The non-transitory computer-readable medium of claim 7, wherein the virtual biocompatible environment is provided at least in part using virtual reality, augmented reality, or augmented reality.

11. The non-transitory computer-readable medium of claim 9 , wherein the visual stimuli include at least one of a scene image and an avatar guide.

12. The non-transitory computer-readable medium of claim 7 , wherein the virtual biocompatible environment is automatically modified in real-time or near real-time.

13. 8. The non-transitory computer-readable medium of claim 7, wherein modifying the virtual bio-compatible environment includes modifying at least one of an audio type, an audio pitch, an audio volume, a scene type, a scene brightness, and a scent.

14. A presentation device; At least one processor; Memory and 11. A system comprising: the memory storing instructions that, when executed by the at least one processor, cause the at least one processor to: providing a virtual biocompatible environment for experience by a user from said presentation device; receiving sensor data associated with the user from the presentation device; analyzing the sensor data using a machine learning model to determine one or more changes in a user state; providing a modified virtual biocompatible environment based at least in part on the analyzed sensor data; and A system that allows the user to:

15. The instructions, when executed by the at least one processor, cause the at least one processor to: determining, based at least in part on the analyzed sensor data, that at least a subset of the sensor data is below a threshold level; providing the modified virtual biocompatible environment based at least in part on the subset of the sensor data being below the threshold level; and The system of claim 14 , further comprising:

16. The system of claim 14 , wherein the virtual biocompatible environment includes at least one of audio stimuli, visual stimuli, and olfactory stimuli.

17. The system of claim 14 , wherein the virtual biocompatible environment is provided at least in part using virtual reality, augmented reality, or augmented reality.

18. The system of claim 16 , wherein the visual stimuli include at least one of a scene image and an avatar guide.

19. The system of claim 14 , wherein the virtual biocompatible environment is automatically modified in real-time or near real-time.

20. 15. The system of claim 14, wherein modifying the virtual biocompatible environment includes modifying at least one of an audio type, an audio pitch, an audio volume, a scene type, a scene brightness, and a scent.

Citation Information

Patent Citations

  • Augmented reality display device with deep learning sensors

    US20180053056A1

  • Virtual reality therapeutic systems

    WO2020254127A1