Systems and methods for enhanced neurological rehabilitation - Patents.com

Abstract

A system and method for patient augmented neurological rehabilitation (ANR) is disclosed. An ANR system generates rhythmic auditory stimuli (RAS) and visual augmented reality (AR) scenes that are synchronized according to a common beat tempo and output to the patient during a treatment session. Patient-worn sensors capture biomechanical data associated with repetitive movements performed by the patient in synchronization with the AR visual content and RAS. A critical thinking algorithm analyzes the sensor data to determine the spatial and temporal relationship of the patient's movements to the visual and audio elements and determine the patient's entrainment level and progress toward clinical / training goals. Furthermore, a 3D AR modeling module configures a processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the determined entrainment level and whether the training goals have been achieved.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 054,599 to McCarthy et al., entitled "Systems and Methods for Augmented Neurological Rehabilitation," filed July 21, 2020, and which also claims priority to U.S. Provisional Patent Application No. 62 / 322,504, entitled "Systems and Methods for Neurological Rehabilitation," filed April 14, 2016, which is a continuation-in-part of U.S. Patent Application No. 16 / 569,388 to McCarthy et al., entitled "Systems and Methods for Neurological Rehabilitation," issued October 22, 2019, which claims priority to U.S. Provisional Patent Application No. 62 / 322,504, entitled "Systems and Methods for Neurological Rehabilitation," filed April 14, 2016, and which is a continuation-in-part of U.S. Patent Application No. 10,448,888, entitled "Systems and Methods for Neurological Rehabilitation," issued October 22, 2019, each of which is incorporated herein by reference as if set forth in its entirety.

[0002] The present disclosure relates generally to systems and methods for the rehabilitation of users with physical disabilities through the administration of music therapy. [Background technology]

[0003] Many controlled studies over the last decade have emphasized the clinical role of music in neurological rehabilitation. For example, it is known that rigorously controlled music therapy can directly enhance cognition, movement and language. The process of listening to music enhances brain activity in many ways, stimulating extensive bidirectional networks between brain regions associated with attention, semantic processing, memory, cognition, motor function and emotional processing.

[0004] Clinical data supports that music therapy enhances memory, attention, executive function, and mood. PET scan studies of neural mechanisms with music in the background have revealed that pleasant music can stimulate an extensive network between cortical and subcortical regions, including the ventral striatum, nucleus accumbens, amygdala, insula, hippocampus, hypothalamus, ventral tegmental area, anterior cingulate area, orbitofrontal cortex, and ventromedial prefrontal cortex. The ventral tegmental area produces dopamine and is directly connected to the amygdala, hippocampus, anterior cingulate area, and prefrontal cortex. This mesocorticolimbic system, which can be activated by music, plays a key role in mediating arousal, emotion, reward, memory attention, and executive function.

[0005] Neuroscience research has revealed how the fundamental organizational processes that form memories in music share mechanisms with non-musical memory processes. The fundamentals of phrase categorization, hierarchical abstraction, and musical patterns have direct parallels with the principles of temporal chunking in non-musical memory processes. This suggests that memory processes activated by music may replace and strengthen non-musical memory processes.

[0006] Thus, there remains a need for improved apparatus, systems, and methods for protecting the use of a user's identity and providing personal information securely. Summary of the Invention

[0007] In one aspect of the disclosed subject matter, a system for enhanced neurological rehabilitation of a patient is provided, the system comprising a computer system having a processor configured with software modules including machine-readable instructions stored on a non-transitory storage medium.

[0008] The software modules include an AA / AR modeling module that, when executed by the processor, configures the processor to generate augmented reality (AR) visual content and rhythmic auditory stimuli (RAS) for output to a patient during a therapy session. Specifically, the RAS includes a beat signal output at a beat tempo, and the AR visual content includes visual elements that move in a prescribed spatial and temporal order based on the beat tempo.

[0009] The system further comprises an input interface in communication with the processor for receiving real-time patient data including time-stamped biomechanical data of the patient associated with repetitive movements performed by the patient in sync with the AR visual content and the RAS, wherein the biomechanical data is measured using a sensor associated with the patient.

[0010] The software module further includes a critical thinking algorithm (CTA) module that configures the processor to analyze the time-stamped biomechanical data to determine a temporal relationship of the patient's repetitive movements to the visual elements and the beat signal output at the beat tempo and to determine an entrainment level to the target parameters, and the AA / AR modeling module further configures the processor to synchronize and dynamically adjust the AR visuals and the RAS output to the patient based on the determined entrainment level.

[0011] According to a further aspect, there is provided a method for enhanced neurological rehabilitation of a patient with a physical disability, the method being implemented on a computer system having a physical processor configured with machine-readable instructions that, when executed, perform the method.

[0012] The method includes providing rhythmic auditory stimuli (RAS) that are output to the patient via an audio output device during a treatment session. Specifically, the RAS includes a beat signal that is output at a beat tempo.

[0013] The method also includes generating augmented reality (AR) visual content for output to the patient via the AR display device. Specifically, the AR visual content includes visual elements that move in a prescribed spatial and temporal order based on the beat tempo and are output in synchronization with the RAS. The method further includes instructing the patient to perform repetitive movements in synchronization with the beat signal of the RAS and the corresponding movement of the visual elements of the AR visual content.

[0014] The method further includes receiving patient data in real time, the patient data including time-stamped biomechanical data of the patient associated with repetitive movements performed by the patient in sync with the AR visual content and the RAS, wherein the biomechanical data is measured using a sensor associated with the patient.

[0015] The method further includes analyzing the time-stamped biomechanical data to determine a temporal relationship of the patient's repetitive movements to the visual elements and the beat signal output in response to the beat signal to determine an entrainment margin. The method further includes dynamically adjusting the AR visual content and the RAS output to the patient to be synchronized and based on the determined entrainment margin not meeting the predefined entrainment margin, and continuing the treatment session with the adjusted AR visual content and the RAS. [Brief description of the drawings]

[0016] The foregoing and other objects, features, and advantages of the devices, systems, and methods described herein will become apparent from the following description of specific embodiments thereof, which are illustrated in the accompanying drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the devices, systems, and methods described herein.

[0017] [Figure 1] FIG. 1 illustrates a system for treating a user by administering music therapy according to an exemplary embodiment of the disclosed subject matter. [Diagram 2]FIG. 1 illustrates some components of a system for rehabilitating a user by providing music therapy, according to an example embodiment of the disclosed subject matter. [Diagram 3] FIG. 1 is a circuit diagram of a sensor for measuring biomechanical motion of a patient according to an exemplary embodiment of the disclosed subject matter. [Figure 4] FIG. 1 illustrates several components of a system in accordance with an exemplary embodiment of the disclosed subject matter. [Diagram 5] 1 illustrates an example representation of components of a system for rehabilitating a user by administering music therapy according to an example embodiment of the disclosed subject matter. [Figure 6] FIG. 1 is a flow diagram of one implementation of an analysis process in accordance with an example embodiment of the disclosed subject matter. [Figure 7] FIG. 1 is a flow diagram of one implementation of a process in accordance with an example embodiment of the disclosed subject matter. [Figure 8] FIG. 1 is a flow diagram of one implementation of a process in accordance with an example embodiment of the disclosed subject matter. [Figure 9] FIG. 1 is a flow diagram of one implementation of a process in accordance with an example embodiment of the disclosed subject matter. [Figure 10] FIG. 1 is a flow diagram of one implementation of a process in accordance with an example embodiment of the disclosed subject matter. [Figure 11] 1 is a time graph illustrating music and a patient's body movements according to an exemplary embodiment of the disclosed subject matter; [Figure 12] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 13] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 14] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 15] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 16] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 17] 13 illustrates a patient's response according to an exemplary embodiment of the disclosed subject matter. [Figure 18] 1 illustrates one implementation of a technique for training a patient to walk, according to an example embodiment of the disclosed subject matter. [Figure 19] 1 illustrates one implementation of a technique for training a patient to ignore, according to an example embodiment of the disclosed subject matter. [Figure 20] 1 illustrates one implementation of a patient intonation training technique according to an example embodiment of the disclosed subject matter. [Figure 21] 1 illustrates one implementation of a technique for musical stimulation training of a patient, according to an example embodiment of the disclosed subject matter. [Figure 22] 1 illustrates one implementation of a technique for gross motor training of a patient, according to an example embodiment of the disclosed subject matter. [Figure 23] 1 illustrates one implementation of a technique for training a patient's grip strength, according to an example embodiment of the disclosed subject matter. [Figure 24] 1 illustrates one implementation of a patient speech cue training technique according to an example embodiment of the disclosed subject matter. [Diagram 25] 1 illustrates one implementation of a minimally conscious patient training technique according to an example embodiment of the disclosed subject matter. [Figure 26] 1 illustrates one implementation of a patient attention training technique according to an example embodiment of the disclosed subject matter. [Figure 27] 1 illustrates one implementation of a patient attention training technique according to an example embodiment of the disclosed subject matter. [Figure 28] 1 illustrates one implementation of a patient attention training technique according to an example embodiment of the disclosed subject matter. [Figure 29] 1 illustrates one implementation of a patient dexterity training technique according to an example embodiment of the disclosed subject matter. [Diagram 30] 1 illustrates one implementation of a technique for oral motor training of a patient according to an example embodiment of the disclosed subject matter. [Diagram 31] 1 illustrates one implementation of a technique for training a patient to breathe, according to an example embodiment of the disclosed subject matter. [Diagram 32]FIG. 1 illustrates a system for enhanced neurological rehabilitation, restoration, or maintenance ("ANR") for administering therapy to a patient according to an exemplary embodiment of the disclosed subject matter. [Diagram 33] FIG. 33 is a graphical visualization of parameters measured during a treatment session performed using the ANR system of FIG. 32, system response, and target / goal parameters according to an exemplary embodiment of the disclosed subject matter. [Diagram 34] 33 is a graph depicting exemplary results relating to metabolic changes resulting from a training session conducted using the ANR system of FIG. 32 in accordance with an exemplary embodiment of the disclosed subject matter. [Diagram 35] 1 is an exemplary augmented reality (AR) display generated by an ANR system for display to a patient during a treatment session according to an exemplary embodiment of the disclosed subject matter. [Figure 36A] 1 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session according to an exemplary embodiment of the disclosed subject matter. [Figure 36B] 1 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session according to an exemplary embodiment of the disclosed subject matter. [Figure 37] 1 illustrates one implementation of a technique for gait training by providing enhanced audio and visual stimuli to a patient, according to an exemplary embodiment of the disclosed subject matter. [Figure 38] FIG. 1 is a mixed system and process diagram conceptually illustrating an ANR system configured to perform a gait training regimen, in accordance with an exemplary embodiment of the disclosed subject matter. [Figure 39] FIG. 39 is a mixed system and process diagram conceptually illustrating components of an Augmented Audio (AA) device of the ANR system of FIG. 38 in accordance with an example embodiment of the disclosed subject matter. [Diagram 40] 1 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session according to an exemplary embodiment of the disclosed subject matter. [Diagram 41]1 is an exemplary AR display generated by an ANR system for display to a patient during a treatment session according to an exemplary embodiment of the disclosed subject matter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention relates generally to systems, methods, and apparatus for implementing a dynamic closed-loop rehabilitation platform system that observes and directs human behavioral and functional changes in language, movement, and cognition that are temporally triggered by musical rhythmic, harmonic, melodic, and intensity cues.

[0019] In various embodiments of the present invention, a dynamic closed-loop rehabilitation platform music therapy system 100 is provided, as shown in FIG. 1, which includes a sensor element and sensor system 102, an edge processing element 104, a collection element 106, an analysis system 108, and a music therapy center 110. As described in more detail below, the sensor element, edge processing element, collection element machine learning process, and music therapy center can be provided on various hardware elements. For example, in one embodiment, the sensor element and edge processing element may be attached to or worn by the patient. In such an embodiment, the collection element and music therapy center may be provided on a portable device. In such an embodiment, the analysis system may be located on a remote server.

[0020] Sensor Systems Throughout this description, the term "patient" is used to refer to the individual receiving music therapy treatment. The term "therapist" is used to refer to the individual administering the music therapy treatment. In some embodiments, a patient can interact with the system described herein without the presence of a therapist administering the treatment.

[0021] The sensor elements 102 provide detected biomechanical data about the patient. In some embodiments, the sensor elements may include (1) a wearable wireless real-time motion detection device, i.e., an IMU (Inertial Measurement Unit), (2) a wearable wireless real-time combined multi-zone plantar pressure / 6-dimensional motion capture (IMU) device, such as sensor 200, (3) a wearable wireless real-time electromyogram (EMG) device, such as sensor 208, and (4) a real-time wireless near-infrared (NIR) video capture device, such as imager 206 (see FIG. 4).

[0022] As shown in FIG. 2, the systems and methods described herein are used in conjunction with treating a patient's gait disorder. Thus, an exemplary sensor 200 can be a combined multi-zone plantar pressure / 6-DOF motion capture device. The sensor 200 records the patient's foot pressure and 6-DOF motion profile while the patient is walking during a music therapy session. In some embodiments, the foot pressure / 6-DOF motion capture device has a variable recording time interval with a sampling rate of 100 Hz for a foot pressure profile that includes 1-4 zones, resulting in 100-400 pressure data points per foot per second.

[0023] The sensor 200 may include foot pressure pads 202 ranging from heel pads (to measure one zone of pressure, e.g., heel strike pressure) to full insole pads (to measure four zones of pressure). Pressure measurements are made by detecting resistance changes in the transducer material as a result of compression due to the patient's weight transferred to the foot. These foot pressure maps are obtained at each sampling interval or at specific points in time during a music therapy session.

[0024] The sensor 200 measures three-dimensional linear acceleration A x , A y , A zThe system may include a 6-D motion capture device 204 that detects changes in movement via 6-degree-of-freedom Micro-Electro-Mechanical System (MEMS)-based sensors that measure vertical and rotational movements such as pitch, yaw, and roll. Sampling at 100 Hz produces 600 movement data points per second. Such foot motion capture is obtained at each sampling interval or at specific points in time during the music therapy session.

[0025] Multi-zone pressure sensing using a six degree of freedom motion capture device allows tracking of spatial and temporal gait dynamics that can be mapped during walking. A circuit diagram of the sensor 200 is shown in FIG.

[0026] As shown in FIG. 4, from a system perspective, patient P uses two foot sensors 200, one for each foot designated right 200R and left 200L. In an exemplary embodiment, right foot sensor 200R wirelessly transmits time-stamped internal measurement device data and heel strike pressure data on a first channel, e.g., channel 5 in the IEEE 802.15.4 Direct Sequence Spread Spectrum (DSSS) RF band. Left foot sensor 200L wirelessly transmits time-stamped internal measurement device data and heel strike pressure data on a second channel, e.g., channel 6 in the IEEE 802.15.4 Direct Sequence Spread Spectrum (DSSS) RF band. As described below, a tablet or laptop 220, optionally used by therapist T, includes a wireless USB hub that includes two IEEE 802.15.4 DSSS RF transceivers tuned to a first and second channel, e.g., channel 5 and channel 6, to capture right / left foot sensor RF data. A handheld wireless trigger 250 is used to start and stop the video and / or to mark and index the time stream, as discussed in more detail below.

[0027] The video analytics domain can be used to extract patient semantic and event information about the therapy session. Patient actions and interactions are components in the therapy that affect the status and management of the therapy. In some embodiments, one or more imaging devices 206 (see FIG. 4 ), such as video cameras, are used with time-synchronized video feeds. While suitable video can be incorporated into the system to capture the patient's movements, near-infrared (NIR) video capture is beneficial to protect patient privacy and reduce video data to process. The NIR video capture device captures NIR video images of the patient's body, such as the patient's torso and limb regions. In addition, the NIR video capture device captures real-time dynamic gait characteristics of the patient as a function of the music therapy session. In some embodiments, the video is taken with a stationary camera, and the background is subtracted and foreground pixels are segmented in the video.

[0028] As shown in Figure 4, when a treatment session begins, one or more video cameras 206 are activated by a tablet or laptop application. The video cameras 206 can be stopped or activated by the therapist with a handheld wireless trigger unit 250. This allows for the creation of labeled and time-stamped indexes of the captured biomechanical sensor data and video data streams.

[0029] In some embodiments, a wearable wireless real-time electromyography (EMG) device 208 can be worn by the patient. EMG sensors provide a full bipedal gait profile of the firing of key muscles for locomotion. Such sensors provide data on the exact time when muscles fire.

[0030] Edge Processing In some embodiments, edge processing occurs at the sensor 200, with sensor data captured from IMUs and pressure sensors. The sensor data is filtered and grouped into various array sizes and further processed into frames reflecting extracted attributes and features, which are transmitted, for example wirelessly, to a collector 106 on a tablet or laptop. Alternatively, it will be appreciated that raw biomechanical sensor data from the sensor 200 can be transferred to a remote processor for collection to perform edge processing functions.

[0031] The wearable sensors 200, 208 and the video capture device 206 generate a sensor data stream that is collectively processed to facilitate biomechanical feature extraction and classification. Sensor fusion combines the output from multiple sensors capturing a common event to capture the outcome better than any single constituent sensor input.

[0032] Capturing patient activity in a music therapy context formalizes the interactions applied to music therapy and in developing patient-specific and generalized formal indicators of music therapy performance and effectiveness. Extracting and analyzing video features allows capturing semantic, high-level information about patient behavior.

[0033] When processing the video, a learned background subtraction method is used to create a background model that accounts for variations in lighting conditions and occlusions in the physical location where music therapy is performed. The result of the background subtraction is a binary map of the foreground with a set of blobs in the foreground that are 2D contours. Thus, the video is segmented into separate image frames for future image processing and sensor fusion. Video information is provided along with additional metadata by fusing edge-processed sensor data from IMUs, foot pressure pads, and EMG sensors. Sensor data can be time-synchronized with other data using an RF trigger. Data can be sent directly to the collector, stored in internal board memory, or analyzed at the edge running the OpenCV library.

[0034] The edge processor 104 can be a microprocessor, such as a 32-bit microprocessor integrated into a foot pressure / 6 DOF motion capture device that enables rapid multi-zone scanning at rates of 100-400 complete foot pressure / 6 DOF movement profiles per second.

[0035] The foot pressure / 6DOF motion capture device collects foot pressure / 6DOF movement profile data for real-time gait analysis, resulting in feature extraction and classification. In some embodiments, the foot pressure / 6DOF motion capture device initializes the microcontroller unit (MCU), continuous operator processing (COP), general purpose input / output (GPIO), serial peripheral interface (SPI), and interrupt requests (IRQ), and sets the desired RF transceiver clock frequency by calling routines including Microcontroller Unit Initialization (MCUInit), General Purpose Input / Output Initialization (GPIOInit), Serial Peripheral Interface Initialization (SPIInit), Interrupt Request Acknowledge Initialization (IRQInit), Interrupt Request Acknowledge (IRQACK), Serial Peripheral Interface Driver Read (SPIDrvRead), and IRQPinEnable. MCUInit is a master initialization routine that stops the MCU watchdog and sets the timer modules to use the bus clock (BUSCLK) as a reference with a prescale of 32.

[0036] The state variable gu8RTxMode is set to SYSTEM_RESET_MODE and the routines GPIOLinit, SPIInit, and IRQInit are called. The state variable gu8RTxMode is set to RF_TRANSCEIVER_RESET_MODE and IRQFLAG is checked to see if an IRQ is asserted. The RF transceiver interrupt is first cleared using SPIDrvRead and the RF transceiver is checked for an ATTN IRQ interrupt. Finally, for MCUInit, a call is made to PLMEPhyReset to reset the physical MAC layer, IRQACK (to acknowledge any pending IRQ interrupts) and IRQPinEnable to pin Enable, IE and IRQ CLR on the negative edge of the signal.

[0037] The foot pressure / 6 DOF motion sensor 200 waits for a response from the foot pressure / 6 DOF motion collection node, for example 250 milliseconds, to determine whether a default full foot pressure scan is performed or whether a mapped foot pressure scan is initiated. In the case of a mapped foot pressure scan, the foot pressure / 6 DOF motion collection node transmits foot pressure scan mapping configuration data to the appropriate electrodes.

[0038] One aspect of the analysis pipeline is a feature set engineering process that defines those captured sensor values ​​and resulting sensor fused values ​​that are used to create feature vectors and define the input data structure for analysis. Representative values ​​are Ax(i), Ay(i), Az(i), Ph(i), where i is the i-th sample, Ax(i) is the acceleration in the x-direction that is lateral to the foot sensor, Ay(i) is the acceleration in the y-direction that is forward to the foot sensor, Az(i) is the acceleration in the z-direction that is upward to the foot sensor, and Ph(i) is the heel strike pressure. The sensor values ​​are shown in Table 1. [Table 1]

[0039] In some embodiments, the sensor fusion approach uses the heel strike pressure value Ph(i) to "gate" the analysis of the following example features to derive a window of data as described below. For example, as shown in Table 2 below, "onset" can be determined based on heel pressure exceeding a threshold signifying heel strike, and "stop" is determined based on heel pressure dropping below a threshold signifying heel off. It is understood that heel strike pressure is one example of a parameter that can be used for "gating" analysis. In some embodiments, "gating" is determined using IMU sensor data, video data, and / or EMG data. [Table 2]

[0040] Higher level features are calculated from the fused sensor values, such as the example values ​​shown in Table 3. [Table 3]

[0041] The system described herein provides the ability to "gate" or provide a "window" to a patient's biomechanical data. Gating of biomechanical data is beneficial for repetitive patient motions, such as repetitive strides while the patient is walking. Sensor data from one or more sources, such as pressure sensors, IMU sensors, video data, EMG data, etc., are used to identify cycles of motion that repeat over time. For example, when a patient walks, foot pressure repeatedly increases and decreases as the patient's foot contacts the ground and then lifts off the ground. Similarly, foot velocity increases as the foot advances forward and decreases to zero while the foot is on the ground. As a further example, the Y position, or height, of the patient's foot cycles between a low position (on the ground) and a high position (approximately in the middle of the stride). The "gating" technique identifies repeating cycles, or "windows," in such data. When a patient is walking, a cycle is repeated with each step. Although there may be variation between cycles, e.g., between steps, a certain pattern is repeated with each cycle. Selecting the onset time (start time) of each cycle involves locating an identifiable point (maximum or minimum) of a biomechanical parameter. The parameter selection for the onset time is selected based on available data. Thus, in some embodiments, the moment when the heel strike pressure exceeds a threshold can be used to define the boundary of the onset time of each cycle. (See, e.g., FIG. 5. The pressures 316a, 316b include periodic characteristics. An "onset" can be determined as the moment when the pressure exceeds a threshold.) Similarly, the onset time can be defined as when the foot velocity drops to zero.

[0042] In some embodiments, the raw frame data is pre-processed to take instantaneous data and "gate", e.g., identify a window, and analyze the data within the window to identify outliers and analyze the data, e.g., exponential analysis, averaging the data over multiple windows. The fusion of sensor data, including both IMU data and heel-strike pressure data, allows for more accurate identification of the onset time of a stride or other repeated motor unit than with data from a single sensor. The sensor data captured during a stride is considered a "window", and information extracted from this analysis includes, e.g., stride length, step number, cadence, time when the step occurred, distance traveled, stance / swing phase, double support time, velocity, symmetry analysis (e.g., between left and right legs), outward swing, shuffling, output vector, lateral acceleration, step width, variability of each of these magnitudes, and further parameters derived from the above information. The feature extraction can be processed on a microprocessor chip, e.g., a 32-bit chip. Wireless, synchronous, gated biomechanical sensor data capture, as well as video data capture capabilities, allow for the creation of time series templates.

[0043] Data can be indexed by the patient or therapist during a music therapy session. The “gating” feature described above is useful for linking specific strides or steps to exception conditions. For example, a therapist may observe a specific exception condition or behavior (such as an anomaly or incident) in the patient's movement. The indexing feature allows the therapist to initiate a capture to “record” the exception condition or behavior via a user interface on a portable tablet or laptop, such as the wireless trigger unit 250 shown in FIG. 4 or via voice control. Annotations can be made that include timestamps and comments such as the patient “stumbled” while walking. Such indexing facilitates the creation of time series templates. These time series templates are studied for the exploration of therapy session events and for the development of time series templates for training machine learning algorithms, such as nonlinear multilayer perceptrons (NLMLPs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs) with long short-term memory (LSTMs).

[0044] In one embodiment, a communication protocol is provided for transferring sensor data (e.g., at the sensor 200) from the edge processing 104 to the collector 106. See Table 4 below. In some embodiments, the RF times out if the connection is idle for more than 100 ms.

[0045] [Table 4]

[0046] In one embodiment, scanning of foot pressure sensor zones is performed by a FootScan routine, in which a FootDataBufferIndex is initialized and foot pressure sensor zones are activated by enabling the MCU direction mode for the output [PTCDD_PTCDDN=Output] and pulling the associated port line low [PTCD_PTCD6=0]. Once a foot pressure sensor zone is activated based on the foot pressure zone scan map, the foot pressure sensor zones connected to the MCU's analog signal ports are sampled and the current voltage measurements are converted into digital form (time zone foot pressure).

[0047] Several variables such as FootDataBufferIndex and IMUBufferIndex are used to prepare an IEEE 802.15.4 RF packet, gsTxPacket.gau8TxDataBuffer[], for transmitting the data used in FootDataBuffer[] and IMUBuffer[]. The RF packet is sent using the RFSendRequest (and gsTxPacket) routine, which checks to see if gu8RTxMode is set to IDLE_Mode and uses gsTxPacket as a pointer to call the RAMDrvWriteTx routine, which calls SPIDrvRead to read the contents of the RF transceiver's TX packet length register. With these contents, the mask length setting is updated, the CRC is added by 2, and the code byte is added by 2.

[0048] SPISendChar is called to send the second code byte, a 0x7E byte, then SPIWaitTransferDone is called again to verify that the transmission has taken place. Once these code bytes have been sent, the remainder of the packet is sent using a for loop, where psTxPkt→u8DataLength+1 is the number of iterations for the sequential set of SPISendChar, SPIWaitTransferDone, and SPIClearRecieveDataReg. Once complete, the RF transceiver is loaded with the packet to send. The ANTENNA_SWITCH is set to transmit, the LNA_ON mode is enabled, and finally a call is made to RTXENAssert to actually send the packet.

[0049] Collector The main function of the collector 106 is to capture data from the edge processing 104 and forward the data to the analysis system 108, receive processed data from the analysis system 108 and forward the data to the music therapy center 110, as described below. In some embodiments, the collector 106 has a control function, e.g., a user interface for logging in, configuring the system, and interacting with the user, and a display unit for visualizing / displaying the data. The collector 106 can include lightweight analytics or machine learning algorithms for classification (e.g., lateral tremor, asymmetry, instability, etc.).

[0050] The collector 106 receives body, movement, and localization data from the edge processor 104. The data received at the collector 106 can be raw or can be processed at the edge 104 before forwarding to the collector. For example, the collector 106 receives fused sensor data, where "windowing" and feature extraction are performed. The forwarded data can include two levels of data, namely, (1) RF packets transmitted from the right / left foot sensors as described in Table 1, and (2) RF packets from the right / left foot sensors containing higher level attributes and features as described in Tables 2 and 3. The collector 106 stores the data locally. In some embodiments, the collector 106 classifies the movement from the received data, e.g., compares the data to a locally stored model (pre-downloaded from the analysis system) or a model transmitted to the analysis system to classify the data. The collector can include a display unit for visualizing / displaying the data.

[0051] In some embodiments, the collector 106 operates on a local computer with a memory, a processor, and a display. Exemplary devices on which the collector may be installed include an augmented reality (AR) device, a virtual reality (VR) device, a tablet, a mobile device, a notebook computer, a desktop computer, and the like. FIG. 2 shows a mobile device 220 having a display 222 and performing the collection function. In some embodiments, the connection parameters for transferring data between the patient sensor and the collector are provided including using Windows Device Manager (e.g., baud rate: 38400, data bits: 8, parity: none, stop bits: 1). In some embodiments, the collector 106 comprises a processor carried or attached to the music therapy patient. In some embodiments, the collector 106 comprises a processor that is remote from the music therapy patient, carried by the therapist, and connected to the music therapy patient via a wireless or wired connection.

[0052] In one embodiment, the foot pressure / 6 DOF movement collection node captures RF transmitted data packets containing real-time foot pressure / 6 DOF movement profile data from the foot pressure / 6 DOF motion capture device. This is initiated by the foot pressure / 6 DOF movement collection node creating an RF packet receive queue driven by a callback function for the RF transceiver packet receive interrupt.

[0053] When an RF packet is received from a foot pressure / 6 DOF motion capture device 200, a check is first made to determine if it is from a new foot pressure / 6 DOF motion capture device or an existing one. If it is from an existing foot pressure / 6 DOF motion capture device, the RF packet sequence number is checked to determine continuous synchronization before further analyzing the packet. If this is a 6 DOF motion capture device capturing foot pressure, a foot pressure / 6 DOF motion capture device context state block is created and initialized. The context state block contains information, e.g., a foot pressure profile.

[0054] On top of this RF packet session level process for inter-node communication is the analysis of the RF packet data payload, which contains a foot pressure profile based on the current variable pressure following six degree of freedom movement. This is constructed as follows: |0x10|Start|F1|F2|F3|F4|Ax|Ay|Az|Pi|Yi|Ri|XOR checksum|

[0055] The IEEE 802.15.4 standard specifies a maximum packet size of 127 bytes, with 47 bytes reserved for Time Synchronous Mesh Protocol (TSMP) operations, leaving 80 bytes for payload. IEEE 802.15.4 is compatible with radio frequency (RF) transceivers in the 2.4 GHz Industrial, Scientific and Medical (ISM) band.

[0056] The RF module contains a complete 802.15.4 physical layer (PHY) modem designed for the IEEE 802.15.4 wireless standard, supporting peer-to-peer, star, and mesh networks. The RF module combines with the MCU to create the required wireless RF data links and networks. The IEEE 802.15.4 transceiver supports 250kbps O-QPSK data on a 5.0MHz channel and full spread spectrum encoding / decoding.

[0057] In some embodiments, control, status reads, data writes, and data reads are accomplished through the detection system node device's RF transceiver interface port. The detection system node device's MPU accesses the detection system node device's RF transceiver through interface "transactions" in which multiple byte-long bursts of data are sent on the interface bus. Each transaction is three or more bursts long, depending on the type of transaction. A transaction is always a read or write access to a register address. The associated data for any one register access is always 16 bits long.

[0058] In some embodiments, control and data transfer for the foot pressure / 6 DOF motion collection node RF transceiver is accomplished using a serial peripheral interface (SPI). While the normal SPI protocol is based on 8-bit transfers, the foot pressure / 6 DOF motion collection node RF transceiver imposes a higher level transaction protocol based on multiple 8-bit transfers per transaction. One SPI read or write transaction consists of an 8-bit header transfer followed by two 8-bit data transfers.

[0059] The header indicates the type of access and the register address. The following bytes are the data to be read or written. SPI also supports recursive "data burst" transactions where further data transfers may occur. The recursive mode is primarily intended for packet RAM access and fast configuration of RF for foot pressure / 6 DOF movement collection nodes.

[0060] In some embodiments, all foot pressure sensor zones are scanned continuously and the entire process is repeated until the sensor is in a reset state or in an inactive power down mode. The 6 degrees of freedom motion is captured by a serial UART interface from the MCU to the Inertial Measurement Unit (IMU). The sampling rate for all sensing dimensions, i.e. Ax, Ay, Az, Pitch, Yaw, Roll, is 100Hz-300Hz and the sampled data is stored in the IMUBuffer[].

[0061] SPIDryWrite is called to update the TX packet length field. Next, SPIClearRecieveStatReg is called to clear the status register, followed by SPIClearRecieveDataReg to clear the receive data register, leaving the SPI interface ready for a read or write. Once the SPI interface is ready, SPISendChar is called to send a 0xFF character representing the first code byte, and SPIWaitTransferDone is called to verify that the transfer has occurred.

[0062] 5 is an example output 300 that may be provided on the display 222 of a handheld device. For example, if treatment is administered to a patient walking, the display output 300 may include a portion 302 for the right foot and a portion 304 for the left foot. As a function of time, the display for the right foot shows acceleration A x 310a, A y 312a, and A z 314a and foot pressure 316a. Similarly, the display for the left foot includes acceleration A x 310a, Ay 312a, and A z 314a and foot pressure 316a.

[0063] Classification is understood to be the correlation of data (e.g., sensor-fused data, feature data, or attribute data) to real-world events (e.g., patient activity or disposition). Typically, classifications are created and performed on the analysis system 108. In some embodiments, the collector 106 has a local copy of some "templates." Thus, incoming sensor data and feature-extracted data can be classified at the collector or analysis system.

[0064] Context refers to the circumstances or facts that form the background of an event, statement, situation, or idea. Context-aware algorithms examine the who, what, when, and where as they relate to the environment and time in which the algorithm is executed on specific data. Some context-aware actions include the identity, location, time, and activity being performed. In using context information to create deterministic actions, a contextual interface is created between the patient, the environment, and the music therapy session.

[0065] The context of the patient's response to the music therapy session may include layers of algorithms that interpret the fused sensor data to infer higher level information. These algorithms refine the context of the patient's response. For example, the patient's biomechanical gait sequence is analyzed as it relates to a particular portion of the music therapy session. In one example, "lateral tremor" is the classifier of interest. Thus, it is determined that the patient's gait will be smoother with less lateral tremor.

[0066] Analysis System The analysis system 108, sometimes referred to as a back-end system, stores the large scale models / archives and includes machine learning / analytics processing using the models described herein. In some embodiments, a web interface for login and dashboards to view the stored data are also provided. In some embodiments, the analysis system 108 is located on a remote server computer that receives data from the collector 106 operating on a portable unit such as a handheld device or tablet 220. It is contemplated that the processing power required to perform the analysis and machine learning functions of the analysis system 108 may also be located on the handheld device 220.

[0067] The data is transferred from the collector 106 to the analysis system 108 for analysis processing. As shown in FIG. 6, the analysis process 400 includes a user interface 402 that receives data from the collector 106. A database storage device 404 receives input data from the collector 106 for storage. Training data and output of the analysis process (e.g., the ensemble machine learning system 410) can also be stored in the storage device 404 to facilitate the creation and refinement of predictive models and classifiers. A data bus 406 allows data to flow through the analysis process. A training process 408 is performed on the training data to derive one or more predictive models. The ensemble machine learning system 410 utilizes the predictive models. The output of the ensemble machine learning system 410 is an aggregation of these predictive models. This aggregated output is also used in classification requests using template classifiers 412 such as tremor, symmetry, smoothness, or learned biomechanical parameters such as entrainment, onset, etc. An API 418 connects to the collector and / or the music therapy center. Treatment algorithms 414 and prediction algorithms 416 include multi-layer perceptron neural networks, hidden Markov models, radial basis function networks, Bayesian estimation models, and the like.

[0068] An example application of the systems and methods described herein is the analysis of a patient's biomechanical gait. Gait sequences are feature extracted to extract a set of characteristic features. The presence of these and other features in the captured sensor-fused data informs a context detection algorithm whether the patient's biomechanical gait sequence is plausible. Robust context detection is required to capture the biomechanical gait sequence, which is then abstracted to a representative population of music therapy patients.

[0069] One example of such an activity is the location of a patient at a given time and the patient's response to music therapy at that time. Identifying and correlating a patient's response to music therapy allows for the recognition of specific patterns of music therapy patient response. By creating a baseline of music therapy patient response and correlating this baseline to future music therapy patient responses, a particular music therapy management regime can be evaluated and analyzed for performance and effectiveness.

[0070] Distance measures along with movement detection along with biomechanical capture of gait are used to determine the trajectory of the patient's path with temporal and spatial variation / deviation between two or more music therapy sessions. From this sensor-fused data capture, features are extracted and classified to label various key patient responses to the therapy. Further analysis of the sensor-fused data using histograms allows for the detection of early music therapy response patterns.

[0071] Analysis of the sensor-fused data from music therapy sessions begins with a patient-specific Bayesian inference model utilizing a Markov chain. The states of the chain represent the patient-specific response patterns captured from a baseline music therapy session. Inferences are made based on knowledge of the occurrence of the patient's response patterns at each sample interval and their temporal relationship to previous states.

[0072] The prediction routine and the multi-layer perceptron neural network (MLPNN) use a directed graph node-based model with a top-layer root node that predicts the necessary conditions to reach a subsequent node to obtain a feature vector of the patient's sensor-fused data. This feature vector of the sensor-fused data includes time-series processed motion data, music signature data, and video image data that are of particular interest for further processing. In this case, the directed graph looks like an upside-down tree with the leaves at the bottom of the tree and the root at the root node. From each node, the routine can proceed left, where left is the left node at the next layer below the top layer where the root node is located, and the left subnode is selected as the next node to be observed, or the routine can proceed right, where right is the right node at the next layer below the top layer where the root node is located, based on the value of a particular variable whose index is stored in the observed node. If the value is less than a threshold, the routine proceeds to the left node, and if it is greater than the threshold, the routine proceeds to the right node. These regions, here left and right, are the prediction space.

[0073] The model uses two types of input variables: ordered and categorical. Ordinal variables are values ​​that are compared to a threshold that is also stored in the node. Categorical variables are discrete values ​​that are tested to see if they belong to a certain limited subset of values, stored in the node. The variables can be applied to different classifications. For example, mild, moderate, and severe can be used to describe tremors, which are examples of categorical variables. Conversely, very finely-spaced ranges of values ​​or numerical scales can also be used to describe tremors (but numerically).

[0074] If the categorical variable belongs to a restricted set of values, the routine proceeds to the left node, otherwise the routine proceeds to the right node. At each node, a pair of entities is used to make this decision: variable_index and decision_rule (threshold / subset). This pair is called a split, and is split according to the variable variable_index.

[0075] Once a node is reached, the value assigned to this node is used as the output of the prediction routine. A multi-layer perceptron neural network is built recursively starting from the root node. All training data, feature vectors, and responses are used to split the root node as described above, and the entities, variable_index, and decision_rule (threshold / subset) partition the prediction space. At each node, an optimal decision rule for the best primary split is found based on a Gini "purity" criterion for classification and the sum of the squared errors of the regression. The Gini coefficient is based on a measure of the total variance across the classes of the set. A small Gini coefficient value indicates that the node contains mainly observations from a single class, which is the desired state.

[0076] Once constructed, a multi-layer perceptron neural network can be pruned using a cross-validation routine. To avoid overfitting the model, some of the branches of the tree are cut off. This routine can be applied to independent decisions. One of the distinguishing properties of the decision algorithm mentioned above (MLPNN) is its ability to calculate the relative decision power and importance of each variable.

[0077] The importance ratings of the variables are used to determine the most frequent types of interactions for the patient interaction feature vector. With the definition of a decision space suitable for distinguishing various categories of responses to music therapy and music therapy events, pattern recognition begins. The decision space can be represented by an N-dimensional graph, where N is the number of attributes or measurements thought to represent responses to music therapy and music therapy events. The N attributes constitute a feature vector or signature that can be plotted on the graph. After sufficient samples have been input, the decision space will reveal collections of responses to music therapy and music therapy events that belong to various categories, and the decision space is used to associate new vectors to these collections.

[0078] The dynamic closed-loop rehabilitation platform music therapy system utilizes several deep learning neural networks to learn and reproduce patterns. In one embodiment, a nonlinear decision space is constructed using an adaptive radial basis function (RBF) model generator. New vectors can be calculated using the RBF model and / or using a k-nearest neighbor classifier. Figure 6 shows the workflow of the machine learning subsystem of the dynamic closed-loop rehabilitation platform music therapy system.

[0079] FIG. 7 illustrates a supervised training process 408 that includes a number of training samples 502, where the inputs are, for example, the features described in Table 3 above, and example outputs are items such as tremor, asymmetry, and force, the magnitude of these items, predictions of change, and a classification of how well the patient is recovering. It is understood that new outputs are learned as part of this process. This process provides a foundation for higher level prediction and classification abstractions as this process is applied to various use cases (e.g., various medical conditions, drug combinations, provider notification, appropriateness, fall prevention). These training samples 502 are run with learning algorithms A1 504a, A2 504b, A3 504c, ..., AN 504n to derive predictive models in M1 506a, M2 506b, M3 506c, ..., MN 506n. Exemplary algorithms include multi-layer perceptron neural networks, hidden Markov models, radial basis function networks, and Bayesian estimation models.

[0080] 8 illustrates an ensemble machine learning system 410 that provides multiple prediction result data 606a, 606b, 606b, ..., 606n as a collection of predictive models M1 506a, M2 506b, M3 506c, ..., MN 506n for sample data 602, e.g., feature extracted data. An aggregation layer 608, e.g., including decision rules and voting, is used to derive an output 610 given multiple predictive models.

[0081] The MR ConvNet system has two layers, the first of which is a convolutional layer that supports average pooling. The second layer of the MR ConvNet system is a fully connected layer that supports multinomial logistic regression. Multinomial logistic regression, also known as softmax, is a generalization of logistic regression to handle many classes (multiclass). In the case of logistic regression, the labels are binary.

[0082] Softmax is a model used to predict the probability of various possible outputs. In what follows, we consider a multi-class classifier with m distinct classes using a softmax final output layer. Y1=Softmax(W11*X1+W12*X2+W13*X3+B1) [1] Y2=Softmax(W21*X1+W22*X2+W23*X3+B2) [2] Y3=Softmax(W31*X1+W32*X2+W33*X3+B3) [3] Ym=Softmax(Wm1*X1+Wm2*X2+Wm3*X3+Bm) [4] generally, Y = softmax(W*X+B) [5] Softmax(X)i = exp(Xi) / (sum of exp(Xj) for j=1 to N) [6] where Y = classifier output, X = sample input (all features scaled (normalized)), W = weight matrix. Classifications could be, for example, score asymmetry, e.g., "moderate asymmetry score, 6 out of 10 (from 10 for high level asymmetry to 0 for no asymmetry)", gait smoothness ("gait smoothness score 8 out of 10 (normal)"), etc. The analysis pipeline is shown in Figure 9.

[0083] Softmax regression allows us to handle more than two classes. In logistic regression, P(x)=1 / (1+exp(-Wx)) where W contains the model parameters trained to minimize the cost function, x is the input feature vector, ((x(1), y(1)),...,(x(i), y(i))) [7] represents the training set. In multi-class classification, softmax regression is used, where instead of 1 and 0 in the binary case, there are N possible distinct values ​​representing classes. Hence, for a training set ((x(1),y(1)),...,(x(i),y(i))), y(n) can be any value ranging from 1 to N classes.

[0084] Then p(y=N|x;W) is the probability for each value of i=1,...,N. Below is the mathematical description of the softmax regression process: Y(x =(p(y=1|x;W),p(y=2|x;W),...,p(y=N|x;W)) [8] Here, Y(x) is the answer to the hypothesis, and given the input x, it outputs a normalized probability distribution for all classes whose sum equals 1.

[0085] The MR ConvNet system convolves all biomechanical data frames in a window as vectors with all biomechanical template filters as vectors to generate a response using an average pooling function that averages the feature responses. The convolution process adds an arbitrary bias to calculate Wx, which is then fed into a logistic regression (sigmoid) function.

[0086] Then, in the second layer of the MR ConvNet system, the subsampled biomechanical template filter responses are translated into a second-order matrix, where each column represents a windowed biomechanical data frame as a vector. Here, a softmax regression activation process is initiated using Y(x)=(1 / (exp(Wx)+exp(Wx)+···+exp(Wx))*(exp(Wx),exp(Wx),···,(exp(Wx)) [9]

[0087] The MR ConvNet system is trained using an optimization algorithm, gradient descent, where a cost function J(W) is defined and minimized. J(W)=1 / j*((H(t(j=1),p(y=1|x;W)+H(t(j=2),p(y=2|x;W)+···+H(t(j),p(y=N|x;W))

[10] where t(j) is the target class. This averages all the cross-entropies over the j training samples. The cross-entropy function is H(t(j),p(y=N|x;W)=-t(j=1)*log(p(y=1|x;W))+t(j=2)*log(p(y=2|x;W))+···+t(j )*p(y=N|x;W)

[11]

[0088] In FIG. 10, the ensemble machine learning system 408 includes multiple predictive models, e.g., template sequence 1 (tremor) 706a, template sequence 2 (symmetry) 706b, template sequence 3 (smoothness) 706c, ..., further templates (other learned biomechanical parameters, e.g., synchronization, onset) 706n, which are applied to trained inputs 702, which may be, e.g., right and left stride length features (x1, x2), right and left stride length variance features (x3, x4), right and left cadence (x6, x7), right and left cadence variance (x8, x9), etc., where the samples (x1, x2, ..., xn) are referred to as vectors X that are input to 702 in the ensemble of ML algorithms. These are trained, reference-normalized, and / or scaled inputs. The aggregate classifier 708 outputs such information as a tremor measure, a symmetry measure, a smoothness measure, etc.

[0089] Music Therapy Center The music therapy center 110 is a decision-making system running on a processing device, such as the handheld device or notebook computer 220 of Figure 2. The music therapy center 110 receives inputs from the sensor data that has been feature extracted in the collector 106, compares them to a prescribed process for administering therapy, and provides auditory stimulation content that is played via the music distribution system 230.

[0090] Embodiments of the present invention use contextual information to determine why certain situations are occurring and encode observed behaviors, thereby generating closed-loop, dynamic, coordinated changes to the system state, and ultimately the music therapy session.

[0091] The interaction between the patient and the music therapy session provides real-time data to determine the patient's awareness of the music therapy context, including movement, posture, stride, and gait response. After the input data is collected by the detection nodes (at the sensors), the embedded nodes process the context-aware data (at the edge processing) to provide real-time dynamic action and / or transmit the data to an analysis system 108, e.g., an elastic network-based processing cloud environment for storage and further processing and analysis.

[0092] Based on the input, the program takes any existing song content and modifies intonation, major / minor chords, time signature, and musical cues (e.g., melodic, harmonic, rhythmic, intensity). The system can overlay a metronome onto an existing song. Song content can be beat mapped (e.g., W in response to an AV or MP3 file) or in MIDI format, so that entrainment margins can be calculated with precise knowledge of when the beats occur. Sensors attached to the patient can be configured to provide haptic / vibration feedback pulses in the musical content. EXAMPLES

[0093] An exemplary application of the method is described herein. Gait training analyzes the real-time relationship between the beats of music being played for the patient and the individual steps taken by the patient in response to those particular beats of music. As described above, gating analysis is used to determine a window of repeated data with some variability with each step or repetitive movement. In some embodiments, the beginning of the window is determined as the time when heel strike pressure exceeds a threshold (or other sensor parameter). FIG. 11 is an exemplary time graph showing the beats of music ("time beats") and the steps taken by the patient ("time steps"). Thus, in this case, the onset time is associated with the "time steps". Specifically, the graph shows a time beat 1101 of music at the time of time beat 1. After some time, the patient takes a step, time step 1102, in response to time beat 1001. The entrainment room 1103 represents the delay, if any, between time beat 1 and time step 1.

[0094] 12-13 show examples of entrainment of a patient's gait using the system described herein. FIG. 12 shows "perfect" entrainment, e.g., zero entrainment margin at all times. This occurs when there is no or very little delay between the time beat and the associated time step taken in response to the time beat. FIG. 13 shows phase-shift entrainment, e.g., where the entrainment margin is non-zero but remains constant or varies minimally over time. This occurs when there is a constant delay over time between the time beat and the time step within an acceptable range.

[0095] With continued reference to FIG. 11, the EP ratio is calculated as the ratio of the time between time beats to the time between time steps. EP ratio = (time beat 2 – time beat 1) / (time step 2 – time step 1) [6] Here, Time Beat 1 (1101) corresponds to the time of the first music beat, and Time Step 1 (1102) corresponds to the time of the patient's step in response to Time Beat 1. Time Beat 2 (1106) corresponds to the time of the second music beat, and Time Step 2 (1108) corresponds to the time of the patient's step in response to Time Beat 2. The goal is EP Ratio=1, or EP Ratio / Coefficient=1. The coefficient is determined as follows:

number

[0096] Figures 14 and 15 show the entrainment response of a patient over time using the techniques described herein. Figure 14 (left Y-axis: EP ratio, right Y-axis: beats per minute, X-axis: time) shows scattered dots 1402 representing the average walking EP ratio of a first patient. The graph shows an upper limit 1404 of +0.1 and a lower limit 1406 of -0.1. Line 1408 shows tempo over time (starting at 60 beats per minute) and increasing in steps to 100 bpm. Figure 14 shows that the EP ratio remains near 1 (±0.1) as the tempo is increased from 60 bpm to 100 bpm. Figure 15 shows the walking EP ratio of a second patient and shows that the EP ratio also remains near 1 (±0.1) as the tempo is increased from 60 bpm to 100 bpm.

[0097] 16 and 17 (Y-axis: entrainment room, X-axis: time) show two patients' responses to changes in time beat (e.g., tempo change) and / or chord changes, changes in haptic feedback, changes in foot cues (e.g., left-right or right-left-cane cues), etc. FIG. 16 shows a time-based graph where the patient's gait is equilibrated with "perfect entrainment" (always zero entrainment room, or very little entrainment room) or with a certain phase-shift entrainment room. As shown, it takes some time (prime time 1602) to equilibrate. FIG. 17 shows a time-based graph where the patient's gait is not equilibrated, e.g., does not reach perfect entrainment or a certain phase-shift entrainment room after a change to the time beat. Prime time is useful because it represents a data set independent of the measurement of entrainment accuracy. The prime time parameter can also be used to screen future songs for suitability. For example, if a patient exhibits a longer prime time when a song is used, such a song may not have much therapeutic potential.

[0098] FIG. 18 illustrates a useful technique for gait training, where repetitive motion refers to the multiple steps a patient takes while walking. Gait training is tailored to each patient population, diagnosis, and condition to provide a personalized music intervention. Based on the input, the program changes the content, intonation, major / minor chords, time signature, and musical cues (e.g., melodic cues, harmonic cues, intensity cues) when applicable. The program can select music using date of birth, listed music preferences, and entrained tempo to provide a passive music playlist to be used daily. Important inputs for gait training are physical activity, e.g., cadence, symmetry, and stride length of the user performing the walking. The program uses connected hardware to provide haptic / vibration feedback at the BPM of the music. Suitable populations for gait training include patients with traumatic brain injury (TBI), stroke, Parkinson's disease, MS, and aging.

[0099] The method begins at step 1802. At step 1804, biomechanical data based on data from sensors, e.g., sensors 200, 206, 208, is received at collector 106. The biomechanical data includes data on onset, stride length, cadence, symmetry, assistive devices, or a collection of other such features of the patient stored or generated by analysis system 108. Exemplary biomechanical data parameters are listed above in Tables 1, 2, and 3. A baseline condition is determined from one or more data sources. First, a patient's gait is detected without music being played. The sensor data and feature data, such as data on the patient's onset, stride length, cadence, symmetry, assistive devices, etc., constitute baseline biomechanical data for the patient at the treatment session. Second, sensor data from previous sessions of the same patient and higher level classification data from analysis system 108 constitute the patient's historical data. Third, the sensor data and higher level classification data for other similarly situated patients constitute population data. Thus, the baseline condition may include data from one or more of: (a) the patient's baseline biomechanical data at the treatment session; (b) data from the patient's previous sessions; and (c) population data. An underlying beat tempo is then selected from the baseline condition. For example, the underlying beat tempo may be selected to match the patient's current cadence prior to the playing of the music. Alternatively, the underlying beat tempo may be selected as a fraction or multiple of the patient's current cadence. As another option, the underlying tempo may be selected to match the underlying beat tempo used in previous sessions of the same patient. As yet another option, the underlying beat tempo may be selected based on the underlying beat tempo used in other patients with similar physical conditions. Finally, the underlying beat tempo may be selected based on any combination of the above-mentioned data. A target beat tempo may also be determined from this data. For example, the target beat tempo may be selected as a percentage increase in the underlying beat tempo with reference to improvements shown by other similarly situated patients. Tempo is understood to refer to the frequency of the beats in the music.

[0100] In step 1806, music provided to the patient from the portable device 220 on the music delivery device 230 (e.g., earphones, headphones, or speakers) is initiated at a base tempo, or a subdivision of the base tempo. To provide music to the patient at the base tempo, music having a constant base tempo is selected from a database, or existing music is modified, e.g., selectively accelerated or slowed down, to provide a beat signal at a constant tempo.

[0101] In step 1808, the patient is instructed to listen to the beat of the music. In step 1810, the patient is instructed to walk at the underlying beat tempo and optionally receive cues for the left and right foot. The patient is instructed to walk "in time" with the beat tempo, for example, so that each step precisely matches the beat of the music. Steps 1806, 1808, and 1810 may be initiated by the therapist or by audio or visual instructions on the portable device 220.

[0102] In step 1812, sensors 200, 206, 208 attached to the patient are used to record patient data such as heel strike pressure, six-dimensional motion, EMG activity, and video recordings of the patient's movements. All sensor data is time-stamped. Data analysis, including the "gated" analysis discussed herein, is performed on the time-stamped sensor data. For example, analysis of the sensor data, such as heel strike pressure, is performed to determine the onset time of each step. Further data received includes the time associated with each beat signal of the music presented to the patient.

[0103] In step 1814, a connection is made to a tuned model (e.g., the ensemble machine learning system 410 of the analysis system 108, or a model downloaded on the collector 106 and running on the portable device 220) for prediction and classification. (It is understood that such a connection may exist pre-existing or may be initiated at this point.) Such a connection typically occurs very quickly or instantly.

[0104] In step 1816, an optional entrainment analysis is applied to the sensor data, performed by analysis system 108. The entrainment analysis includes determining the delay between the beat signal and the onset of each step taken by the patient. The output from the entrainment analysis is a determination of the accuracy of the entrainment, e.g., a measurement of the instantaneous relationship between the underlying tempo and the patient's steps, as described above with respect to entrainment margin and EP ratio. If the entrainment is not accurate, e.g., if the entrainment margin is not acceptably constant, then in step 1818, adjustments are made, e.g., speeding up or slowing down the beat tempo, increasing the volume, increasing the sensory input, overlaying a metronome or other relevant sound, etc. If the entrainment is accurate, e.g., if the entrainment margin is acceptably constant, then in step 1820, an incremental change is made to the tempo. For example, the underlying tempo of the music played on the portable device is increased, e.g., by 5%, toward the target tempo.

[0105] At step 1822, a connection is made to an entrainment model for prediction and classification. (It is understood that such a connection may be pre-existing or may be initiated at this point.) At step 1824, an optional symmetry analysis is applied to the sensor data. As an output of the symmetry analysis, a determination is made as to the symmetry of the patient's gait, e.g., how closely the movement of the patient's left leg matches the movement of the patient's right leg in stride length, speed, stance phase, swing phase, etc. If the steps are not symmetrical, e.g., below a threshold, at step 1826, an adjustment is made to the music transmitted to the patient by the portable device. A first correction can be made to the music played while one of the patient's legs is moving, and a second correction can be made to the music played while the other of the patient's legs is moving. For example, a minor chord (or increased volume, sensory input, tempo change, or sound / metronome superimposition) can be played on one side, e.g., the affected side, and a major chord can be played on the other side, e.g., the non-affected side. The machine learning system 410 can predict when a symmetry problem is about to occur based on a "fingerprint" that leads to a symmetry problem, for example by analyzing movements that indicate asymmetry. Asymmetry can be determined by comparing someone's normal gait parameters to their background to determine how affected their side is compared to the other side.

[0106] At step 1828, a connection is made to an entrainment model for prediction and classification. (It is understood that such connections may be pre-existing or may be initiated at this point.) At step 1830, an optional analysis of center of balance, e.g., whether the patient is leaning over, is performed on the sensor data. The analysis may be performed by combining the output of multiple foot sensors, and the video output. As an output from the center of balance analysis, a determination is made as to whether the patient is leaning over. If the patient is leaning over, at step 1832, a cue is issued to the patient to stand up straight, provided by the therapist or by audio or visual instructions on the handheld device.

[0107] At step 1834, a connection is made to the entrainment model for prediction and classification. (It is understood that such connections may be pre-existing or may be initiated at this point.) At step 1836, an onset analysis is applied to the sensor data, for example indicating that the patient is hesitant or having difficulty initiating walking. As an output from the onset analysis, a determination is made as to whether the patient is experiencing problems initiating. If the patient is experiencing problems initiating, for example below a threshold, haptic feedback may be provided to the patient, which may include a countdown at the beat tempo or prior to the start of the song at step 1838.

[0108] At step 1840, it is optionally determined whether the patient is using an assistive device, e.g., a cane, crutches, walker, etc. In some embodiments, the portable device 220 provides a user interface for the patient or therapist to input information regarding the use of the assistive device. If a cane is present, the analysis changes to three beats, e.g., cane, right foot, left foot, and cues are provided at step 1842 with "left foot," "right foot," and "cane" provided by the therapist or by audio or visual instructions on the portable device.

[0109] In step 1844, a connection is made to the entrainment model for prediction and classification. (It is understood that such a connection may exist pre-existing or may be initiated at this point.) An optional entrainment analysis 1846 is applied to the sensor data generally as described above in step 1816, with the differences described below. For example, the entrainment may be compared to previous entrainment data from an earlier time point in the session, previous entrainment data from a previous session with the patient, or data related to entrainment of other patients. As an output from the entrainment analysis, the accuracy of the entrainment is determined, e.g., how closely the patient's gait matches the underlying tempo. If the entrainment is not accurate, in step 1848, adjustments are made generally as described above in step 1818.

[0110] If the entrainment is accurate, in step 1850 it is determined whether the patient is walking at the target tempo. If the target tempo is not reached, the method proceeds to step 1820 (described above) where incremental changes are made to the tempo. For example, the underlying tempo of the music played on the portable device is increased or decreased, for example by 5%, towards the target tempo. If the target tempo is reached, the patient may continue therapy for the remainder of the session (step 1852). In step 1854, music at the desired tempo may be curated and left on the device 220 of FIG. 2 for use outside of therapy sessions. This music content is used by the patient as a challenge / practice between dedicated therapy sessions. In step 827 the program ends.

[0111] 18 may be performed in an order different from that disclosed. For example, the assessments in steps 1816, 1824, 1830, 1836, 1840, 1846, and 1850 may be performed simultaneously. Furthermore, multiple connections to analysis system 108 (e.g., steps 1814, 1822, 1828, 1834, and 1844) may be performed only once throughout the described treatment session.

[0112] FIG. 19 shows a technique useful in ignore training. In ignore training, the systems and methods described herein use connected hardware to provide tactile / vibration feedback when the patient accurately hits the target. The connected hardware includes a device, a video motion capture system, or a connected bell. All of these devices are connected into the described system and have a speaker that vibrates when tapped and plays auditory feedback. For example, a connected bell provides data to the system in the same way as the sensor 200, e.g., data about the patient hitting the bell. A video motion capture system provides video data to the system in the same way as the video camera 206. An important input in ignore training is information about tracking of movement to a specific location. The program uses connected hardware to provide tactile / vibration feedback when the patient accurately hits the target. A suitable population for ignore training includes patients with spatial neglect or unilateral visual neglect conditions.

[0113] The flow diagram of the ignore training shown in FIG. 19 is almost the same as the flow of the locomotor training shown in FIG. 18, with the differences described below. For example, a baseline test defines the patient's condition and / or improvement from a previous test. In some embodiments, the baseline test includes showing four objects spaced equally from left to right on a screen, for example, on the display 222 of the portable device 220. The patient is instructed to tap the objects to the beat of the background music by a cue displayed on the display 222 or verbally by the therapist. As with the locomotor training, the patient is instructed to tap a bell to the beat of the background music. Feedback is given for each accurate tap. Once the baseline information has been collected, a number of objects spaced equally from left to right are displayed on the screen. As described above, the patient is instructed to tap the objects in order from left to right to the beat of the background music. Feedback is given for each accurate tap. Similar to locomotor training, analysis system 108 evaluates the patient's response, classifies the response, and provides instructions to add or remove objects, or to speed up or slow down the tempo of the music to reach a target tempo.

[0114] FIG. 20 illustrates an approach useful for intonation training. In intonation training, the systems and methods described herein rely on speech processing algorithms. The phrases typically chosen are common words in the bilabial, guttural, and vowel categories. Hardware is connected to the patient to provide haptic feedback in beats per minute to one of the patient's hands. The key inputs for intonation training are voice timbre, spoken words, and speech rhythm. Populations suitable for intonation training include patients with Broca's aphasia, expressive aphasia, non-fluent aphasia, apraxia, autism spectrum disorder, and Down's syndrome.

[0115] The flow diagram of intonation training shown in FIG. 20 is almost the same as that of gait training shown in FIG. 18, with the following differences. For example, haptic feedback is provided to one of the patient's hands to prompt tapping. The patient is then instructed to listen to music being played with cues displayed on the display 222 or verbally by the therapist. A spoken phrase to be learned is played back separated into two parts, the first of the two parts being a higher pitch and the second of the two parts being a lower pitch. The patient is then instructed to sing the phrase using the device with the two pitches being played with cues displayed on the display 222 or verbally by the therapist. As with gait training, the analysis system 108 evaluates the patient's response, classifies the response in terms of pitch accuracy, words spoken, ranking by the patient or assistant / therapist, provides instructions to provide alternative phrases, and compares the response with the target speech parameters.

[0116] FIG. 21 illustrates an approach useful in musically stimulated training. In musically stimulated training, the systems and methods described herein rely on speech processing algorithms. Familiar songs are used along with algorithms that isolate expected parts (called expectancy violations). The hardware includes speakers that receive and process the singing by the patient, and in some embodiments, the therapist can manually provide input regarding the accuracy of the singing. Important inputs are information related to the timbre of the voice, the words spoken, the rhythm of the speech, and musical preferences. Suitable populations include patients with Broca's aphasia, non-fluent aphasia, TBI, stroke, and primary progressive aphasia.

[0117] The flow diagram of the musical stimulation training shown in FIG. 21 is almost the same as the flow of the locomotor training shown in FIG. 18, with the differences described below. For example, a song is played for the patient, and the patient is instructed to listen to the song by cues displayed on the display 222 or verbally by the therapist. Musical cues are added to the song. Words or sounds are then omitted at expected locations, and gestural musical cues are played to prompt the patient to sing the missing words or sounds. As with the locomotor training, the analysis system 108 evaluates the patient's response, classifies the response in terms of pitch accuracy, words spoken, ranking by the patient or assistant / therapist, and instructs to play further parts of the song to improve the speech to the target speech parameters.

[0118] FIG. 22 illustrates an approach useful in gross motor training. In gross motor training, the systems and methods described herein are targeted to aid in ataxia, range of motion, or initiation. The more difficult parts of the movement are musically "asccented" using, for example, melodic, harmonic, rhythmic, and / or strength cues. The key input is movement-related information in X, Y, Z direction capture using connected hardware or a video camera system. Suitable populations include patients with neurological and orthopedic strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

[0119] The flow diagram for gross motor training shown in Figure 22 is nearly identical to the flow for gait training shown in Figure 18, with the differences described below. As with gait training, the patient is cued to move in time with the beat that is the basis of the musical selection. Analysis system 108 evaluates the patient's response, classifies it in terms of movement accuracy and synchronization as described above, and provides instructions to speed up or slow down the tempo of the music being played.

[0120] FIG. 23 illustrates an approach useful for grip training. For grip training, the systems and methods described herein rely on sensors associated with a gripping device. The hardware includes a gripping device with a pressure sensor and a connected speaker associated with a portable device 220. The key input is pressure exerted by the patient on the gripping device in a manner similar to heel strike pressure measured by the sensor 200. Suitable populations include patients with neurological and orthopedic strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

[0121] The flow diagram for grip training shown in Figure 23 is substantially identical to the flow for gait training shown in Figure 18, with the differences described below. As with gait training, the patient is cued to apply force to the gripping device in time with the beat that is the basis of the musical selection. Analysis system 108 evaluates the patient's response, classifies it in terms of accuracy and synchronization of the movement as described above, and provides instructions to speed up or slow down the tempo of the music being played.

[0122] FIG. 24 illustrates an approach useful for speech cue training. In speech cue training, the systems and methods described herein rely on voice processing algorithms. The hardware may include a speaker that receives and processes the singing by the patient, and in some embodiments, the therapist can manually provide input regarding the accuracy of the speech. Important inputs are voice timbre, words spoken, speech rhythm, and music preferences. Suitable populations include patients with problems with robots, word finding, and speech stuttering.

[0123] The flow diagram for speech cue training shown in Figure 24 is substantially identical to the flow for gait training shown in Figure 18, with the differences described below. As with gait training, the patient is cued by cues displayed on the display 222 or verbally by the therapist to speak sentences, syllables in time with the beat that is the basis for the musical selection. Analysis system 108 evaluates the patient's speech and classifies the response in terms of speech accuracy and synchronization as described above, and provides instructions to speed up or slow down the tempo of the music being played.

[0124] Figure 25 shows an approach that is useful in training minimally conscious patients. The systems and methods described herein rely on an imaging system, such as a 3D camera, to measure the direction the patient is looking if both eyes are open, and the resulting patient pulse or heart rate. The program explores and optimizes heart rate, stimulation, respiratory rate, eye closure, posture, and restlessness. Suitable populations include comatose patients and patients with impaired consciousness.

[0125] The flow diagram of the minimally conscious patient training shown in FIG. 25 is almost the same as the flow of the gait training shown in FIG. 18, with the following differences. As with the gait training, the patient is presented with increasing stimuli at the patient's breathing rate (PBR). For example, the patient is first presented with musical chord stimuli at PBR, and it is observed whether the patient's eyes are open. If the patient's eyes are not open, the stimuli are successively increased, from humming a simple melody at PBR, to singing "ahhh" at PBR, to singing the patient's name (or playing a recording of such a sound) at PBR, and after each input, it is checked whether the patient's eyes are open. The analysis system 108 evaluates the patient's eye tracking, classifies the response in terms of the level of consciousness, and gives instructions to change the stimuli.

[0126] 26-28 show an approach that is useful in attention training. In attention training, the systems and methods described herein operate in a closed loop to help patients sustain, divide, shift, and select attention. No visual cues are allowed to suggest which movement to make. Suitable populations include patients with brain tumors, multiple sclerosis, Parkinson's disease, and neurological diseases and injuries.

[0127] The flow diagram of sustained attention training shown in FIG. 26 is almost the same as the flow of gait training shown in FIG. 18, with the differences described below. Similar to gait training, the patient is given an instrument (e.g., any musical instrument that can be operated, such as drumsticks, drums, keyboard, or wirelessly connected to each) and instructed by cues displayed on the display 222 or verbally by the therapist to proceed along or work on audio cues defined in levels 1 to 9 shown in FIG. 26. The analysis system 108 evaluates the patient's ability to complete the tasks accurately, classifies the response, and changes the tempo or difficulty of the task. Similarly, FIG. 27 shows a flow diagram of attention switching training, in which the patient is instructed by cues displayed on the display 222 or verbally by the therapist to proceed along or work on audio cues alternating between the left and right ears. FIG. 28 shows a flow diagram for divided attention, in which the patient is instructed by cues displayed on the display 222 or verbally by the therapist to proceed along or work on audio cues using audio signals in both the left and right ears.

[0128] The flow diagram of the dexterity training shown in FIG. 29 is almost identical to the flow of the gait training shown in FIG. 18, with the differences described below. In the dexterity training, the patient is instructed to tap the piano keys with his / her fingers to gather a range of basic movement and motor information. A piece is started at a specific number of beats per minute, and the patient begins tapping with the basic number of fingers. The analysis system 108 evaluates the patient's ability to complete the task accurately, classifies the response, and changes the tempo or difficulty of the task.

[0129] The flow diagram of the oral-motor training shown in Figure 30 is almost identical to the flow of the gait training shown in Figure 18, with the differences described below. In the oral-motor training, the patient is instructed to perform a task in which they alternate between making two sounds, e.g., "wah" and "ahh." Analysis system 108 evaluates the patient's ability to complete the task accurately, classifies the response, and changes the tempo or difficulty of the task, e.g., by providing different target sounds.

[0130] The flow diagram of the breathing exercise shown in FIG. 31 is substantially the same as the flow of the walking exercise shown in FIG. 18, with the differences described below. In the breathing exercise, a baseline breathing rate and shallowness are determined. Music is presented at a tempo that is the baseline for the patient's breathing rate, and the patient is instructed to perform the breathing work at the level described in FIG. 31. Analysis system 108 evaluates the patient's ability to complete the work accurately, classifies the response, and changes the tempo or the difficulty of the work, for example by providing a different breathing pattern.

[0131] Further described herein are methods, systems, and devices for using Augmented Reality (AR) and Augmented Audio (AA) to support next generation medical treatment systems for improving or maintaining motor function. Exemplary embodiments of the systems and methods disclosed herein for Augmented Neurological Rehabilitation, Recovery, or Maintenance (ANR) build on the entrainment approach by utilizing additional sensor streams to determine treatment effectiveness and inform closed-loop treatment algorithms. Exemplary systems and methods for neurological rehabilitation that can be utilized to implement embodiments of the ANR system and method are shown and described above, but are also shown and described in co-pending and commonly assigned U.S. patent application Ser. No. 16 / 569,388, entitled "Systems and Methods for Neurological Rehabilitation," to McCarthy et al., which is a continuation of U.S. Provisional Patent Application No. 62 / 322,504, entitled "Systems and Methods for Neurological Rehabilitation," filed April 14, 2016, and which claims priority thereto, and the specifications of each of these patent applications and patents are incorporated herein by reference as if set forth in their entireties.

[0132] According to one or more embodiments, the ANR system and method can include providing an AR 3D dynamic model of specific people or objects, including querying cloud and local databases for images and videos of people, or images. The ANR system and method can also be configured to fuse and / or synchronize the dynamic model, patient or human, audio content, and environmental context into a synchronized state to leverage the neuroscience of musical abilities and how visual imagery impacts recovery (e.g., use of mirror neurons) to improve motor function.

[0133] According to one or more embodiments, the ANR system and method includes a method for combining AA techniques with repetitive motor actions. Augmented Audio (AA) combines real-world sounds with additional computer-generated audio "layers" that enhance sensory input. Neuroscience, with rhythm at its core, uses stimuli to engage the motor system in repetitive motor actions such as walking. Adding AA to treatment protocols can enhance treatment efficacy, increase adherence to treatment protocols, and provide greater safety in the form of increased situational awareness for patients. Disclosed embodiments configured to add AA can leverage the neuroscience of music by mixing many audio signals, including external environmental sound inputs, pre-recorded content, rhythmic content, and audio guidance, in a synchronized state. Additionally, it may have the ability to combine algorithmically generated music with intrinsic rhythmic cues, as detailed in

[0184] , for patients with movement disorders or physical disabilities. This can be done by blending this generated rhythm with input from the patient's real-time biometric data for interactive feedback states. Exemplary systems and methods for algorithmically generating auditory stimuli for use in connection with neurological rehabilitation are shown and described in co-pending and commonly assigned U.S. patent application Ser. No. 16 / 743,946, entitled "Improved Music for Repetitive Movement Exercises," filed Jan. 15, 2020, to McCarthy et al., which is a continuation of U.S. patent application Ser. No. 16 / 044,240, entitled "Improved Music for Repetitive Movement Exercises," filed July 24, 2018 (now U.S. Patent No. 10,556,087, issued February 11, 2020), which claims the benefit of priority under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application No. 62 / 536,264, filed July 24, 2017, each of which is incorporated by reference in its entirety herein as if set forth herein.

[0134] Neuroplasticity, entrainment, and science of mirror neurons are fundamental scientific building blocks that support the disclosed embodiments. Entrainment is the term for the activation of the motor centers of the brain in response to external rhythmic stimuli. Studies have shown that audio-motor pathways exist within the reticulospinal tract, a part of the brain involved in movement. Priming and timing of movements through these pathways has shown the ability of the motor system to couple with the auditory system to trigger movement patterns (Rossignol and Melville, 1976). The entrainment process has been shown to effectively increase walking speed (Cha, 2014), reduce gait variability (Wright, 2016), and lower the risk of falling (Trombetti, 2011). Neuroplasticity refers to the brain's ability to strengthen pre-existing neural connections, allowing individuals to acquire new skills over time. Studies have demonstrated that music can facilitate changes in certain motor areas of the brain, indicating that music can promote neuroplasticity (Moore et al., 2017). Mirror neurons fire both when you perform an action and when you observe another person performing an action. Thus, when you see another person performing an action, your brain responds as if it were you performing that activity. This allows us to learn behaviors by imitation. This is important in the context of the disclosed embodiments because when a patient is observing a simulation of a human in augmented reality, the patient can use mirror neurons to imitate those actions.

[0135] According to one or more embodiments, the ANR system and method are configured to process the image / video to remove and / or add people or objects smaller or larger than a specified size from the image / video in response to patient or therapist exception conditions received as input to the ANR system. Such exceptions may be patient responses such as instructions to reduce scene complexity or therapist instructions to incorporate occlusions, which may be people / objects that increase scene complexity. Embodiments may accommodate recording of all data for all patient or therapist exception conditions in addition to the session data itself.

[0136] According to one or more embodiments, the ANR system and method includes a telepresence approach that allows the system to connect a therapist to a patient in a remote location, including tracking of the patient's biomechanical motion against AR 3D dynamic models of people / objects in addition to fully supporting all the localized features that the patient and therapist would experience if they were in the same location.

[0137] The AR 3D dynamic model is a software-based algorithmic process of the ANR system and method to generate an animated presentation of AR / VR visual scenes to the patient based on fundamental principals of mirror neuron neuroplasticity, entrainment, and science to facilitate outcomes toward clinical or training goals. Telepresence methods are configured to use the present invention to provide the ability to activate an interactive video link between the therapist and the remote patient. Telepresence methods are compatible with using the present invention to project images / videos from the remote patient to show the patient's relative position in the AR 3D dynamic model. Telepresence methods can also provide the ability for the therapist to adjust the AR model in real time (spatial location, or which items are available) to allow modifications to the session. Telepresence methods also allow the therapist to compare the patient to the model.

[0138] FIG. 32 depicts a conceptual overview of the main components of an exemplary ANR system 3200 that uses closed-loop feedback to measure, analyze, and act on a person to facilitate outcomes toward clinical or training goals. It should be understood that the ANR system 3200 can be realized using various hardware and / or software components of the system 100 described above. As shown, the ANR system measures or receives inputs related to gait parameters, environment, context / user intent (including past execution), physiological parameters, and real-time feedback on outcomes (e.g., closed-loop real-time decision making). One or a combination of these inputs can be sent to a clinical thinking algorithm module (CTA) 3208 that controls the analysis of and action on these information. In some embodiments, an example of real-time feedback is the CTA 3208 determining that the measured quality of the user's gait measures (e.g., symmetry, stability, gait cycle time variance) has exceeded a modeled safety threshold, which results in a new cued response. Such cued responses can be visual and audio, for example amplifying the prominence of a musical beat by adding a metronome audio layer to modify the AR scene being modeled, or steer the user towards safer walking speeds and movement behaviors.

[0139] In the CTA module 3208, actions taken by the ANR system are defined based on the analysis of the inputs. The actions can be output to the patient in the form of various stimuli, including music (or other components), rhythm, auralization of sounds, spoken words, augmented reality, virtual reality, augmented audio, or haptic feedback. As shown in FIG. 32, the decisions made by the critical thinking algorithm in relation to various inputs, results, etc. are provided to an AR / AA output modeling module 3210, which is programmed to dynamically generate / modify outputs to the patient. The outputs are provided to the patient via one or more output devices 3220, such as visual and / or audio output devices, and / or haptic feedback devices. For example, AR visual content can be output to AR glasses 3222 worn by the patient. Augmented audio content can be provided to the patient via audio speakers or headphones 3225. As will be appreciated, other suitable visual or audio display devices can be used without departing from the scope of the disclosed embodiments. Additionally, although the various elements of the system are shown separately in FIG. 32, it should be understood that the features and functions of the various aspects of the system can be combined.

[0140] The clinical thinking algorithm module 3208 in this medical / therapeutic system can be configured to execute critical thinking algorithms that emphasize restoring, maintaining, or enhancing motor function, including, but not limited to, upper limbs, lower limbs, sway, postural stability, foot drop, dynamic stability, respiration, mouth movement, breathing, endurance, heart rate training, respiratory frequency, optical flow, boundary support training, strategy training (ankle, knee, hip), attractor binding, muscle firing, training optimization, and gait. Also, one or more of these CTAs can be implemented in synchrony or combination with other interventions that share similar goals. Examples include implementing CTA in conjunction with functional electrical stimulation, deep brain stimulation, transcutaneous electrical nerve stimulation (TENS), gamma frequency audio entrainment (20Hz-50Hz), or other electrical stimulation systems. Additionally, medication and CTA manipulations can be combined with antispasticity medications, or medications can be combined with injections of neurotoxins. For example, CTA can be administered or initiated at times when such interventions have been shown to be maximally effective, or can be used prior to reaching the time point at which such interventions are maximally effective as a method of priming the motor system.

[0141] AANR System Input Inputs to the ANR system 3200 are important in enabling the system to measure, analyze, and operate in a continuous loop to facilitate outcomes toward clinical or training goals.

[0142] Receiving inputs in the system may include using sensors to measure the person's motion to determine biomechanical parameters of the motion (e.g., temporal, spatial, and side-to-side comparison). Such motion sensors may be placed anywhere on the body and may be one sensor or many sensors. Other types of sensors may be used to measure other input parameters, including respiration rate, heart rate, oxygen levels, body temperature, electroencephalogram (EEG) to record spontaneous electrical activity of the brain, electrocardiogram (ECG or EKG) to measure electrical activity of the heart, electromyogram (EMG) to assess and record electrical activity produced by skeletal muscles, photoplethysmography (PPG) to detect changes in blood volume in the microvascular bed of tissue, often using a pulse oximeter to measure changes in light absorption of the skin, optical inertial measurement units, video cameras, microphones, accelerometers, gyroscopes, infrared, ultrasound, radar, RF motion detection, GPS, barometers, RFID, radar, humidity, or other sensors to detect physiological or biomechanical parameters. For example, in the example ANR system 3200 shown in Figure 32, gait parameters can be measured using one or more sensors 3252, such as an IMU, footpad sensors, smartphone sensors (e.g., accelerometers), environmental sensors, etc. Additionally, physiological parameters can be measured using one or more sensors 3254, such as PPG, EMG / EKG, respiration rate sensors, etc.

[0143] Additionally, contextual information regarding desired outcomes, environment of use, data from past sessions, other techniques, and other environmental conditions can be received as inputs to the CTA module to tailor the CTA's response. For example, as shown in FIG. 32, contextual information 3258 and environmental input 3256 information can be received as inputs to further inform the operation of the CTA 3208. An example of the use of contextual information is that historical information about a user's walking patterns can be used in conjunction with an artificial intelligence (AI) or machine learning (ML) system to provide more personalized clinical goals and behaviors for the patient. These goals can also modify goal parameters such as steps per minute, walking speed, heart rate variability, oxygen consumption (VO2 max), respiratory rate, session length, asymmetry, variability, distance walked, or limits on desired heart rate.

[0144] Additionally, Bluetooth Low Energy (BLE) beacons or other wireless proximity technologies such as wireless triangulation and angle of arrival can be used to detect environmental information and facilitate wireless location triggers to make people / objects appear and / or disappear depending on the detected location in the patient's field of view relative to the AR 3D dynamic model. In some embodiments, the AR 3D dynamic model output by the ANR system can be controlled by the therapist and / or beacon triggers to change or maintain the guidance requirements for the patient. These triggers can also be used with the gait or physiological data mentioned above to provide additional triggers in addition to the wireless beacon triggers. For example, gait data feedback from an IMU product enables a gait feedback loop that provides the ANR system 3200 with the ability to make changes in the software process of the AR 3D dynamic model.

[0145] A Clinical Thinking Algorithm According to one or more embodiments, the CTA module 3208 executes clinical thinking algorithms configured to control the applied therapy to facilitate outcomes toward clinical or training goals. Clinical goals may include items such as those described in connection with FIG. 18-FIG. 31, with further examples including interventions for agitation in Alzheimer's disease, dementia, bipolar disorder, and schizophrenia, as well as training / physical activity goals. Various non-limiting example techniques are discussed in this section that can be used to inform appropriate rehabilitation responses determined by CTA, such as adjusting rhythmic tempo or synchronized AR visual scenes. Each of these techniques can also be implemented using CTA alone or in combination with each other. In one or more embodiments, the system 3200 can be configured to combine CTA with entrainment principles for repetitive motor actions, and in other cases, these techniques can be combined with each other toward other goals.

[0146] By way of example, and not by way of limitation, the CTA module 3208 of the ANR system 3200 can be configured to utilize a combination of biomechanical data, physiological data, and context to create a virtual treadmill output via the AR / AA output interface. While the treadmill keeps pace with the physical belt movement for someone, the virtual treadmill is dynamically adjusted using CTA according to entrainment principles to independently regulate the pace of the person's walking or movement, similar to other exercise interventions. However, in addition to or instead of using rhythmic stimulation to drive an individual toward the biomechanical goals described above, the virtual treadmill can be generated based on the patient's entrainment and dynamically controlled toward goal parameters, such as those listed above as goal parameters.

[0147] The goal parameters may also be set based on input from a clinician, user, historical data, or baseline conditions, or to a degree that is clinically meaningful. Further goal parameters may be defined by or follow recommendations for exercise length, duration, and intensity specific to a particular condition, such as cardiac disease, asthma, COPD, fall prevention, musculoskeletal disease, osteoarthritis, general aging, etc.

[0148] Further described herein is an exemplary embodiment of the ANR system 3200, in which the CTA module 3208 is configured to utilize biomechanical data, physiological data, and context to provide gait training therapy in the form of a virtual treadmill and rhythmic auditory stimuli output via the AR / AA output interface 3220.

[0149] FIG. 37 is a process flow diagram illustrating an example routine 3750 for providing gait training therapy to a patient using the ANR system 3200. FIG. 38 is a mixed system and process diagram conceptually illustrating aspects of the ANR system 3200 performing a gait training routine 3750, according to an example embodiment of the disclosed subject matter. As shown, a sensor 3252, specifically a foot-mounted IMU, captures sensor data related to gait parameters that are provided to the CTA module 3208. Additionally, the AA / AR modeling element 3210 with audio engine is configured to receive input from the CTA and generate a cued audio ensemble including one or more of rhythmically cued music content, interactive audio guidance, and spatial audio effect processing. Similarly, an AA / AR modeling element 3210 comprising an AR / VR modeling engine (also referred to as an AR 3D dynamic model) is shown receiving input from the CTA and is configured to generate a cued visual ensemble including one or more of virtual AR characters and objects (e.g., a virtual walking person), background motion animations (e.g., virtual treadmill, steps / footprints, and animations), and scene lighting and shading.

[0150] FIG. 39 is a mixed system and process diagram conceptually illustrating an example audio output device 3225 and augmented audio generating elements of the ANR system 3200 in more detail. In one embodiment, as shown in FIG. 39, the AA device can capture environmental sounds, for example, using a stereo microphone. The AA device can also generate audio output using a stereo transducer. The AA device can also include a head-mounted IMU. As will be appreciated, the AA device can also include hardware and software components for audio signal processing to receive, process, and output the augmented audio content received from the AA / AR module 3210, either alone or with other content such as environmental sounds. As shown, the CTA module 3208 receives gait parameters, including those received from sensors including foot-mounted IMUs and head-mounted IMUs. Additionally, in one embodiment, the CTA receives data related to physiological parameters from other sensor devices such as PPG sensors.

[0151] Returning to Figure 37, in step 3700, the patient wearing the AR / AA output device 3220 and IMU sensor 3252 begins walking as the ANR system 3200 performs calibration and collects preliminary gait data such as stride length, speed, gait cycle time, symmetry, etc. In step 3701, the CTA module 3208 determines an underlying rhythmic tempo for both the music playback and the displayed virtual AR scene. For example, the underlying rhythmic tempo may be determined by CTA, as described in connection with Figure 18.

[0152] Using the user's walking cycle time as input, in step 3702, the audio engine (i.e., the audio modeling component of the AR / AA modeling module 3210) generates corresponding music that is tempo adjusted with auxiliary rhythmic cues, such as a metronome sound, to amplify the prominence of the beat.

[0153] In step 3703, the visual AR engine (i.e., the visual modeling elements of the AR / AA modeling module 3210) generates a moving virtual scene, such as that understood in the video game industry. More specifically, in one embodiment, the virtual scene includes visual elements presented under the control of the CTA and sharing a common reference timing with the audio engine to synchronize the elements of the visual scene with music and rhythmic tempo. Although the AR scene described herein includes a virtual treadmill or a virtual person or footprints, the AR scene can also be any one or more of the various examples described herein, such as a virtual treadmill, a virtual walking person, a virtual crowd, or a dynamic virtual scene.

[0154] In steps 3704 and 3705, music / rhythm and visual content is delivered to the patient using an AR / AA device 3220, such as a lightweight head-up display (e.g., AR goggles 3222) with earphones 3225. Under the control of the CTA, the patient receives therapy instructions in steps 3706 and 3707 via off-screen audio cues generated by a speech engine, which may include previews of pre-ambulatory training to familiarize the patient with the visual scenes and audio experience and allow them to practice.

[0155] FIG. 40 shows an example AR virtual scene 4010 presented to which the patient is entrained. As shown, the scene may include a moving 3D image of another person walking "in front" of the patient, whose steps and walking motions are synchronized to the music tempo. More specifically, in one embodiment, the AR character walks to the same tempo as the underlying beat tempo of the audio content generated by the CTA and the audio engine. In this example, the patient's goal may be for their steps to match rhythmically with the audio and visually with the character. Additionally, as shown in FIG. 40, the AR scene may include multiple footprints with additional cues such as L and R indicating left and right feet.

[0156] The scene including the footsteps may be virtually moving towards the patient at a prescribed speed, while the virtual character is walking ahead of the patient away from the patient. In one embodiment, the scene may be moving based on the walking cycle time, GCT(right foot) / 2*60=music tempo determined by CTA. Furthermore, the additional cues generated in association with the AR scene may include rhythmic audio cues that reinforce the visual cues. For example, one effective reinforcement method may include the AA system 3210 generating sounds of the footsteps of the virtual characters synchronized with the rhythm, simulating a group march to a common beat. In addition to timing the movement of the visual elements with the rhythmic audio stimuli, by providing the patient with rhythmic audio elements generated based on the visual stimuli, the system further reinforces the virtuous cycle in the basic therapeutic concept of entrainment and mirror neurons.

[0157] As a further example, FIG. 41 shows an exemplary AR virtual scene 4110 presented to which the patient entrains. As shown in FIG. 41, a virtual treadmill can be generated and dynamically controlled based on the patient's entrainment to goal parameters, such as those listed above as goal parameters. In this example, the generated AR treadmill animates the movement of the treadmill surface 4115 and generates virtual steps at the same tempo as that prescribed by the CTA for the auditory stimuli. Furthermore, the 3D animation of the virtual treadmill can include visually highlighted steps or tiles that the patient can use as visual targets while simultaneously entraining to the generated rhythm under the control of the CTA. Thus, in this example, the patient's goal can be for their steps to match the audio rhythmically and visually with the animated goal steps. As further shown in FIG. 41, animated footprints 4120 can be shown on the surface of the virtual treadmill 4115 moving in a direction toward the patient, indicated by a directional arrow that can be in the opposite direction if the patient is performing a backward gait training exercise. Additionally, animations highlighting target steps 4125L (left foot step) and / or 4125R (right foot step), respectively, are highlighted in sync with the CTA rhythm to encourage the user to walk with the corresponding foot (e.g., left or right). Additionally, as described further herein, virtual scenes such as those shown in Figures 40-41 can be dynamically adjusted based on the patient's entrainment room (EP) and corresponding changes to the audio stimuli. Other adjustments to the virtual scene may include modifying the virtual background environment to simulate different walking scenarios, weather, surfaces, lighting, and slopes.

[0158] Returning now to FIG. 37, once the patient begins walking, in step 3708, a biomechanical sensor (e.g., sensor 3252) measures real-time data for use in assessing the patient's entrainment level. In step 3709, an entrainment room (e.g., as described in FIG. 18) is determined by the CTA module 3208 and used to determine how the goals of the training session should be achieved. As described in previous embodiments of the present disclosure, the entrainment room can be the basis for modifying the rhythmic audio stimuli and visual scenes, which modification is performed in step 3710. For example, the CTA analyzes the input data history of the patient's walking cycle time compared to the rhythmic interval of the beats delivered to the patient by the audio device. Exemplary techniques for modifying audio stimuli based on entrainment room are similarly described above. In one example, if the EP values ​​calculated for the patient's steps over a period of time are not within a defined EP value tolerance and / or are not sufficiently consistent, the CTA module can instruct the AA / AR modeling module to adjust (e.g., slow down) the tempo of the RAS and adjust the movement speed of the AR scene accordingly to synchronize with the RAS. In a further example, if sufficient step time is in sync with the beat time, the patient is considered to be in sync by the CTA.

[0159] Once entrained, in step 3711, the CTA evaluates whether the patient has reached the goal. If the goal has not been reached, one or more goal parameters may be adjusted in step 3712. For example, in one example, the CTA compares the RAS tempo and associated AR scene rate to a goal tempo parameter (e.g., training goal / therapy goal), and the rhythmic tempo and / or scene movement rate are altered to account for this comparison. Exemplary methods by which the CTA module 3208 may adjust rhythmic auditory stimuli based on entrainment room are shown, for example, in connection with FIG. 18 and described above.

[0160] For example, if the patient has not reached his or her training speed goal, modifying the goal parameters may include increasing or decreasing the music tempo in step 3712, thereby encouraging the patient to walk faster or slower using the mechanism of action of the RAS. Alternatively, another training goal may be to increase the patient's stride length, which may be achieved by slowing the motion speed parameters of the images. By modifying the visual scene, the patient is encouraged to model the presented visual example using the mechanism of mirroring.

[0161] It is important to understand that the audio and visual outputs reinforce each other, and the visual scenes are layered together to synchronize with the rhythmic stimulation. Depending on the program selected (e.g., CTA), the CTA module 3208 dynamically adjusts the visual scenes and rhythmic tempo to achieve the therapeutic goal.

[0162] The above example shows how the ANR system 3200 can use the CTA module 3208 to control the synchronization of music tempo and AR scenes based on biomechanical sensor input to facilitate gait training. It should be appreciated that the principles of this embodiment are applicable to many disease indications and rehabilitation scenarios.

[0163] In an exemplary configuration, a virtual treadmill can be generated by the ANR system 3200 to adjust the patient's gait toward a target parameter of oxygen consumption. In this example, a virtual treadmill is generated and controlled to adjust walking speed toward an oxygen consumption or efficiency target parameter using entrainment. FIG. 33 is a diagrammatic visualization of a real-time session conducted using the ANR system 3200 using V02 max as the target parameter, tempo change as an interim goal, and entrainment to promote physiological changes related to v02 max. FIG. 33 shows an example of how this process works in real-time. More specifically, FIG. 33 is a graphical user interface showing various salient data points and parameter values ​​that are measured, calculated, and / or adjusted in real-time by the ANR system during the session. As shown, the top of the interface shows a graph of the entrainment room values ​​calculated for each step in real-time throughout the session. Specifically, the top bar shows the individual EPs calculated for each step, which in this example is the phase correlation between the step time interval and the beat time interval. The central zone around EP=1 represents steps that are well entrained to the beat, i.e., have an EP value within the prescribed range. The next window below provides a status bar indicating whether the parameters are within safe ranges. The next window below shows the real-time response driven by the CTA based on, among other things, the measured parameters, entrainment, the other previously mentioned inputs, and feedback to the CTA. In particular, the circular icon represents the algorithmic response, which includes both tempo changes and changes in the level (e.g., volume) of the rhythmic stimulation. The bar below it shows only tempo, which changes on its own. The next window below shows the real-time tempo of the rhythmic stimulation delivered to the patient over time according to the CTA response. The bottom window shows the oxygen consumption and target parameters measured over time.Although not shown in FIG. 33, it is understood that the patient may be presented with an augmented reality scene (e.g., a virtual treadmill) with visual elements that are moved in sync with the rhythmic stimuli and dynamically adjusted in sync with adjustments to the real-time tempo of the rhythmic stimuli. An example of how the AA / VR module 3210 may be configured to synchronize the speed of the visual animation and audio may include defining a relationship between the speed of the displayed repetitive motion and the tempo of the audio cue. For example, the relationship between the audio and visual elements is defined by calculating the treadmill speed and step interval based on the beat tempo. Additionally, the treadmill reference position, footstep timing, and beat-synchronized animation are synchronized to the output time of the beat, including the beat tempo. Using time-scale modification and video frame interpolation techniques known in the animation industry, a wide range of synchronized virtual scenes can be programmatically generated on-demand by the AA / VR module 3210 based on the defined relationships between the audio and visual elements.

[0164] FIG. 34 is a graph representing metabolic changes (each represented by a set of two dots connected by a dashed line) during the first training session for seven patients. FIG. 34 shows data supporting the ability of intentional entrainment to improve an individual's oxygen consumption. In this case, the graph shows the oxygen consumption (ml / kg / meter) of a person before and after rhythmic training with the ANR system. The figure shows an average reduction of 10%. This result shows that the entrainment process can improve endurance and reduce energy expenditure while walking.

[0165] This above process can be similarly performed for each of the various possible target parameters discussed above (e.g., oxygen consumption can be replaced with another target such as heart rate) and can be performed for walking or other interventions as discussed in connection with Figures 18 to 31.

[0166] According to one or more embodiments, the ANR system 3200 can be configured to compare real-time measured information about the person's movement with components (e.g., momentary tempo, rhythm, harmony, melody, etc.) of the AR images and / or musical content output to the patient during a treatment session. This can be used by the system to calculate entrainment parameters, determine phase entrainment, or define basic carryover characteristics. For example, the AR images can move at the same speed or pace as the target parameters. Alternatively, the AR-related movement of the images can be entrained to a rhythmic stimulus to synchronize with how the person should move.

[0167] Examples of the output of the AR 3D dynamic model may include projecting a therapist (virtual character) or person (virtual character) walking in the patient's field of view, initiated by the person providing the treatment (real therapist). For example, FIG. 35 shows a figure of a therapist or coach projected in front of the patient or trainer using AR, for example using AR glasses known in the art. This AR 3D dynamic model is controlled using one or various CTAs. In the case of a combination of multiple CTAs, the virtual therapist can start using the technique shown and described in connection with FIG. 22 and then proceed with a gait training therapy as shown and described in connection with FIG. 18. Alternatively, these can be done simultaneously in a dual task. During these cases, the virtual character can be controllably displayed by the system as walking or moving backwards or forwards with a smooth motion similar to the patient's healthy side. This can activate mirror neurons, encouraging neurons on the affected side to mirror the actions of neurons on the healthy side. This process may also include providing audio stimuli and synchronizing the virtual and / or physical person to the stimuli.

[0168] In another example, the AR 3D dynamic model can be configured to simulate a scenario in which the patient is walking through or around a group of people and / or people and objects in front of and / or beside the patient. For example, FIG. 36A shows a group of people projected in front of the patient using AR. The system can be configured to project a group of people or a person moving faster or slower than the person's baseline to prompt the patient to move at a similar speed or stop / start in the real-world environment. The group of people or person can also be synchronized to a rhythmic auditory stimuli beat or another desired target. Various difficulty levels of guidance can be initiated by the AR 3D dynamic model. As should be appreciated, the AR view of the therapist, crowd, people, obstacles, and the like can be dynamically adjusted using the AR 3D dynamic model depending on the output of the CTA.

[0169] In another example, the AR 3D dynamic model can be configured to simulate a scenario in which the patient is walking through or around an array of cones that provide a virtual obstacle course for the patient to navigate. Although cones are common obstacles in a therapeutic environment, other embodiments of this can be configured to simulate normal activities of daily living (e.g., grocery shopping). These cones, along with the virtual obstacles, can encourage lateral stepping to each side and backwards walking to change direction rather than just forward walking. Again, wireless beacon triggers can be used to cause the ANR system to present cones that appear and / or disappear. The beacons are activated based on the person's location being detected relative to the cones. Additionally, different difficulty levels can be initiated in the navigation time and length. The goal parameters in this example can be a measure of walking speed or quality of walking. A successful navigation would be to virtually navigate around the cones without hitting them. The system can be configured to present increasingly challenging levels (e.g., more obstacles and faster speeds) unless the person successfully avoids the obstacles and gait quality deteriorates (as measured by increased variability or worsening asymmetry).

[0170] In another example, the AR 3D dynamic model can be configured to simulate a scenario in which the patient is walking in or around a cave and / or cliff that may include obstacles for a realistic effect. The realism increases the detail required in the navigation over the previously presented use cases. In another example, with a person with an asymmetric gait pattern, a winding path can be presented that requires the person to take longer steps on the affected side. The winding path can also be separate cliffs that the person must straddle to avoid falling. Wireless beacon triggers can be used to cause the ANR system to make cave and / or cliff obstacles appear and / or disappear, resulting in varying difficulty levels in navigation time and path length. Sensor data can be used by the system to synchronize the movement to the winding path. The patient's navigation requirements can be biomechanical responses that direct changes in the basic prescribed course. The system is configured such that wireless space-time beacon triggers affect changes in the AR 3D dynamic model. A temporal aspect of such wireless triggers is the ability to turn them on and off. This allows maximum flexibility in planning the guidance path for the course the patient needs to take as part of the treatment session. The target parameter at this moment can be a measure of walking speed or gait quality. Successful guidance would be to follow the path without veering off or falling off a cliff. The system can be configured to present increasingly more challenging levels (e.g., more obstacles and faster speeds) unless the person successfully stays on the path and the quality of the gait deteriorates (measured by increased variability or worsening asymmetry).

[0171] In another example, the AR 3D dynamic model can be configured to simulate a scenario where the patient is standing or sitting still and is asked to move forward when a virtual object is presented and approaches each foot. For example, FIG. 36B shows a footstep diagram projected in front of the patient using AR. The ANR system can generate a virtual scene where an object can approach the patient's left or right to prompt the patient to step sideways. The object is presented approaching the patient with a predefined tempo or beat, following the decision tree described in FIG. 22. Also, a visual of the correct movement by the therapist or patient from a previous treatment can be projected.

[0172] In another exemplary implementation of the AR 3D dynamic model, the ANR system can be configured to incorporate haptic feedback into the treatment. Haptic feedback can be used by the system to signal, for example, if the user gets too close to objects or people in the projected AR environment. Rhythmic haptic feedback can also be synchronized with auditory cues to amplify the sensory input. AR can also adaptively and individually cue the onset of movement, for example during episodes of freezing of gait in someone with Parkinson's disease.

[0173] In another exemplary implementation of the AR 3D dynamic model, the ANR system can be further configured to incorporate optical head tracking. This tracking can be incorporated as feedback to the ANR system, which is configured to activate auditory input in response to where their eyes or head are pointing. For example, in a person with left-sided neglect who only interacts with the right side of their environment, eye and head tracking can provide input to how much of the environment is engaged in the person's left hemisphere, causing the system to generate auditory cues to direct more attention to the left hemisphere. This data can also be used to track progress over time, as clinical improvement can be measured in terms of the degree of awareness in each hemisphere. Another example of this is people with eye movement disorders, who may improve by having their left-to-right visual scans timed to an external auditory rhythm.

[0174] In another exemplary implementation of the AR 3D dynamic model, the ANR system can be configured to provide a digital presence of past sessions to display the user's improvement. These models can be played back after the session to allow for session-to-session or lifetime treatment comparisons. The digital presence of past sessions (or extended sessions), when combined with audio input of the session, can also be used as a mental imagery task to practice during walking sessions to reduce fatigue. The model can display differences in walking speed, cadence, stride length, and symmetry to help show the user's changes over time and how treatment may improve the user's gait. This presence can also be used by therapists prior to a session to help prepare a training plan or approach for subsequent sessions. The modeled presence can also be used by researchers and clinicians to better visualize the evolution of the patient's progress and again animated in a 3D image.

[0175] In another exemplary implementation of the AR 3D dynamic model, the AR / VR environment synchronized with the music content can create various walking or dancing patterns to include egg-shaped paths, spiral paths, meandering paths, and paths that cross other paths, as well as dual-task walking. Dance rhythms such as tango have been shown to have benefits derived from neurological music therapy (NMT) and the RAS that can be applied to the entire human body.

[0176] According to one or more embodiments, the ANR system can be configured to utilize AA techniques to enhance the entrainment process and provide environmental context to the person to assist with the AR experience. To enhance the recovery process, the system can be configured to generate example AA experiences, as further described herein, based on inputs from the environment, sensor data, the AR environment, entrainment, and other methods.

[0177] An example of AA for a therapeutic / medical use case is addressing safety concerns and reducing risk to patients undergoing exercise therapy. The ANR system can be configured to improve situational awareness while listening to music on headphones by mixing external sounds that momentarily exceed a minimum audio loudness threshold with the rhythmic, audio-cued content of the therapy. Examples of external sounds are the sound of a car horn or an emergency vehicle siren, which are automatically synchronized to interrupt normal auditory stimuli and provide the person with awareness of possible danger. To perform this and other functions, the listening device can have additional microphones and dedicated digital signal processing to perform this task.

[0178] In further embodiments, an ANR system implementing AA can be configured to combine aspects of AA with the manipulation of spatial awareness by lining up rhythmic auditory cues to the patient's "affected side" while the patient is undergoing a walking therapy session. For example, if the patient's right side requires a greater degree of therapy, audio cued content can be lining up spatially to the right for emphasis. Exemplary systems and methods for neurological rehabilitation using techniques for side-specific rhythmic auditory stimulation are disclosed in co-pending and commonly assigned U.S. Patent Application No. 62 / 934,457, entitled "Systems and Methods for Neurological Rehabilitation," filed November 12, 2019, and incorporated herein by reference in its entirety as if set forth herein.

[0179] In further embodiments, an ANR system implementing AA can be configured to provide specific auditory cues to improve head position spatial awareness during gait training, thereby encouraging the user to keep their head up and on centerline with eyes facing forward during the entrainment process or other CTA experience to improve balance and spatial awareness.

[0180] In further embodiments, an ANR system implementing AA can be configured to provide binaural beat sounds and link the beat sounds with human physiology (e.g., respiration rate, electrical brain activity (EEG), heart rate) to improve cognition and enhance memory. The ANR system can be configured to provide binaural beat audio signal inputs that complement the RAS signal inputs. The real-time entrainment and quality of the gait measurements made by the system are similarly complemented by physiological measurements. For example, a system configured for binaural beat audio uses a differential frequency signal output to the left and right ears, the difference being 40 Hz, the "gamma" frequency of neural oscillations. These frequencies can reduce amyloid accumulation in Alzheimer's disease patients and aid cognitive flexibility. By providing such audio signals to the user via an AA device while the user is performing RAS gait training, a second type of neural entrainment can be achieved simultaneously with biomechanical RAS entrainment. The network hypothesis of brain activation suggests that both gait and cognition are affected. Therefore, such auditory sensory stimuli entrain neural oscillations in the brain, and rhythmic auditory stimuli entrain the motor system.

[0181] In further embodiments, an ANR system implementing AA can be configured to provide a phase-coherent sound field (e.g., spatial perspective correct sound) when the patient turns his / her head or changes posture. The sound field is a virtual 3D image created by stereo speakers or headphones. The sound field allows the listener to hear the exact location of the sound source. An example of manipulating the sound field in a therapy session is to keep the virtual coach's voice "in front" of the patient even while the patient's head is turned to the side. This feature can help avoid disorientation, thus creating a more stable, predictable and safe audio experience while performing therapy. This feature can also be combined with an AR virtual coach / therapist in front of the person in FIG. 34. This feature can also be combined with knowledge of the path or direction the person should take in the real world.

[0182] In further embodiments, the ANR system can be configured to combine AA with augmented reality (AR) such that virtual sound effects (e.g., cheers and crowd footsteps) create a sound field that is in phase with the patient's visual gaze when the patient is synchronized with the virtual crowd. This sound can also create a perception of distance or closeness from an object. Changing the spatial location or loudness of sounds in such a way can also be used as a target for targeting with AR and 3D imagery in conjunction.

[0183] In further embodiments, the ANR system can be configured to combine AA with augmented reality (AR) to create a virtual instrument therapy. Musical instruments such as bells, drums, pianos, guitars, etc. can be common training tools for patients. Digital models of these instruments can be created and AA feedback can be given to the patient during interaction, giving them an immersive experience and the perception that they are physically playing the instrument. The difficulty can also be modified to help the patient progress over time and show improvement. Examples of modifications can include adding additional keys on a piano or additional strings on a guitar. In addition to the virtual instruments, virtual musical scores or notations can be displayed in real time while the patient is playing the instrument, either the virtual or real instrument. Other examples can be combined with the concepts discussed in connection with FIG. 19, where the connected hardware can be replaced with AR. Similar logic can be used for other documented interventions.

[0184] In further embodiments, the ANR system can be configured to perform AA in combination with telepresence to provide a spatially accurate audio experience to the therapist. The audio can also be generated to create a perception of distance or closeness from an object. By varying the spatial location or loudness of the AA and leveraging the AR model, the system can be used to more effectively determine whether the patient is achieving the goals associated with playing the virtual device, providing a more accurate and tailored experience to the patient.

[0185] According to one or more embodiments, the ANR system can implement a type of AA, namely, a rhythmic stimulation engine (RSE). The rhythmic stimulation engine is a custom-made rhythmic auditory stimulation that embodies the entrainment principle while generating original, customized, rhythmic auditory content for the patient to promote therapeutic effects. In conditions such as Parkinson's disease, it can also be beneficial to have a constant rhythmic "soundtrack" in the patient's environment. The RSE can also be configured to perform this continuous rhythmic background neural stimulation without the need for access to pre-recorded music. In one example, the ANR system can be configured to perform AA in combination with the rhythmic stimulation engine (RSE) and AR to create a fully synchronized feedback state between the input biodata and the external audio input from the therapeutic environment to the generated rhythmic content, the AR / AA output. In another example, the system can be configured to interactively adjust the tempo of the rhythmic audio content generated by the RSE with the walking cadence of the patient. In another example, the tempo and time signature of the rhythmic audio content generated by the RSE can be interactively adjusted with the accuracy of entrainment and beat factor of the patient-user, such as a patient using a cane or assistive device. In another example, the RSE can provide neurostimulation to enhance the efficacy of gait therapy in combination with assistive technologies such as augmented suits, augmented exoskeletons, and / or FES devices. In another example, the RSE can generate rhythmic audio content from a stored library of traditional dance rhythm templates that can be therapeutically administered to the patient's upper body and limbs. This can also be expanded by combining the AR techniques mentioned above, such as dancing crowds and virtual dance floors. In another example, machine learning techniques, such as self-learning AI and / or rule-based systems, can generate rhythms in real time, governed by inertial measurement unit (IMU) inputs that report cadence and quality of gait parameters such as symmetry and gait cycle time variation.Using unsupervised ML clustering or decision tree models, various walking patterns can serve as input to a generative music system.

[0186] According to one or more embodiments, the ANR system can perform a kind of AA, namely sonification, which means applying different amounts of signal distortion to the music content depending on how close or far the patient is from the target goal. The degree and type of sonification helps to move the patient to a corrective state. The novel combination of sonification and entrainment can provide a feed-forward mechanism for auditory-motor synchrony using entrainment, while also simultaneously providing a feedback mechanism using distortion of the music content for some other biomechanical or physiological parameter that the individual can adjust. For example, the combination of adding signal distortion to the music signal while increasing the volume of rhythmic cues has a greater effect than either method alone.

[0187] According to one or more embodiments, the ANR system can perform CTA in combination with injection of neurotoxin as follows: CTA can also work towards improving motor function, such as walking, by applying the entrainment principle. Neurotoxin injection can also be used to target gait improvement by targeting muscle spasticity. These injections take 2-4 days to take effect and last for up to 90 days (e.g., efficacy period). Administration of CTA for entrainment principle (e.g., setting one or more parameters of CTA) can also be targeted to the efficacy curve of neurotoxin injection, with less intense training during the period before the injection takes effect and intensifying training during the efficacy period.

[0188] According to one or more embodiments, the ANR system can be configured to calculate the entrainment parameters by synchronizing the heart rate or the respiration rate to the music content instead of the biomechanical motion parameters. An example use case is for people with various forms of agitation, such as dementia, Alzheimer's disease, bipolar disorder, schizophrenia, etc. In this use case, the basic parameters can be determined by the heart rate or the respiration rate. The entrainment or gradual entrainment can be determined by comparing the music content with the heart rate or the respiration rate. Furthermore, the goal can be set to reduce the degree of agitation to improve the quality of life of such people.

[0189] As can be seen from the foregoing description, a system for enhanced neurological rehabilitation of a patient, according to one or more embodiments, may include one or more of the following items.

[0190] A computer system having one or more physical processors is configured with software modules including machine-readable instructions. The software modules may include a 3D AR modeling module that, when executed by the processor, configures the processor to generate and present augmented reality visual and audio content to a patient during a treatment session. The content includes visual elements that move in a prescribed spatial and temporal order and rhythmic audio elements that are output at a beat tempo.

[0191] The computer system also includes an input interface in communication with the processor for receiving input including time-stamped biomechanical data of the patient related to movements performed by the patient in association with the AR visual and audio content, and physiological parameters measured using one or more sensors associated with the patient.

[0192] The software module also includes a critical thinking algorithm that configures the processor to analyze the time-stamped biomechanical data to determine spatial and temporal relationships of the patient's motion to the visual and audio elements and determine an entrainment level of the patient to the target physiological parameter. Additionally, the 3D AR modeling module further configures the processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the determined entrainment level to the target parameter.

[0193] The above-mentioned systems, devices, methods, processes, and the like can be implemented using hardware, software, or a combination thereof suitable for the application. Hardware can include general purpose computers and / or special purpose computing devices. This includes implementations in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices or processing circuits, with internal and / or external memory. Also, or alternatively, this can include one or more application specific integrated circuits, programmable gate arrays, programmable array logic components, or one or more other devices configurable to process electronic signals. It will be further understood that implementations of the above-mentioned processes or devices can include computer executable code written using a structured programming language such as C, an object-oriented programming language such as C++, or any other high-level or low-level programming language (including assembly languages, hardware description languages, database programming languages ​​and techniques), which can be stored and compiled or interpreted on one of the above-mentioned devices, and heterogeneous combinations of processors, processor architectures, or various hardware and software combinations. In another aspect, the method can be embodied in a system that performs its steps, and can be distributed in some manner across multiple devices. Additionally, processing may be distributed across multiple devices, such as the various systems described above, or all functionality may be integrated into a dedicated stand-alone device or other hardware. In alternative aspects, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to be within the scope of the present disclosure.

[0194] The embodiments disclosed herein may include a computer program product including computer executable or computer usable code that performs any and / or all of the steps when executed on one or more computing devices. The code may be non-transiently stored in computer memory, which may be the memory in which the program is executed (such as a random access memory associated with a processor), or a disk drive, flash memory, or any other optical, electromagnetic, magnetic, infrared, or other device, or a combination of these devices. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium that carries / conveys the computer executable code and / or any input or output from the computer executable code.

[0195] It is understood that the above-described devices, systems, and methods are described by way of example, and not by way of limitation. Unless expressly stated otherwise, the disclosed steps can be modified, supplemented, omitted, and / or rearranged without departing from the scope of the present disclosure. Numerous variations, additions, omissions, and other modifications will be apparent to those skilled in the art. In addition, the order or presentation of the method steps in the above description and drawings is not intended to require that the recited steps be performed in that order, unless a certain order is expressly required or a certain order is otherwise apparent from the context.

[0196] The method steps of the implementations described herein are intended to include any manner suitable for performing such method steps consistent with the patentability of the appended claims, unless a different meaning is expressly given or is otherwise apparent from the context. Thus, for example, performing step X includes any manner suitable for causing another party, such as a remote user, a remote processing resource (e.g., a server or cloud computer), a machine, etc. to perform step X. Similarly, performing steps X, Y, and Z may include any manner of directing or controlling any combination of such other individuals or resources to perform steps X, Y, and Z to obtain the benefit of such steps. Thus, the method steps of the implementations described herein are intended to include any manner suitable for causing one or more other parties or entities to perform those steps consistent with the patentability of the appended claims, unless a different meaning is expressly given or is otherwise apparent from the context. Such parties or entities need not be under the direction or control of any other parties or entities, and need not be located within any particular jurisdiction.

[0197] Moreover, it should be understood that the methods described above are provided by way of example, and unless expressly stated otherwise, the disclosed steps can be modified, supplemented, omitted, and / or rearranged without departing from the scope of the present disclosure.

[0198] It is understood that the above-described methods and systems are described by way of example, not by way of limitation. Numerous variations, additions, omissions, and other modifications will be apparent to those skilled in the art. In addition, the order or presentation of method steps in the above description and drawings is not intended to require that the recited steps be performed in that order, unless a particular order is expressly required or otherwise apparent from the context. Thus, while specific embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications in form and detail can be made in those embodiments without departing from the spirit and scope of the present disclosure.

Claims

1. 1. A system for enhanced neurological rehabilitation of a patient, comprising: A computer system having a processor configured with a software module including machine-readable instructions stored in a non-transitory storage medium, the software module comprising: a computer system including an AA / AR modeling module that, when executed by the processor, configures the processor to generate augmented reality (AR) visual content and rhythmic auditory stimuli (RAS) for output to a patient during a therapy session, the RAS including a beat signal output at a beat tempo, and the AR visual content including visual elements that move in a defined spatial and temporal order based on the beat tempo; and an input interface in communication with the processor for receiving real-time patient data including time-stamped biomechanical data of the patient associated with repetitive movements performed by the patient in sync with the AR visual content and the RAS, the biomechanical data being measured using a sensor associated with the patient; Including, The software module further comprises: a critical thinking algorithm (CTA) module that configures the processor to analyze the time-stamped biomechanical data to determine a temporal relationship of the patient's repetitive movements to the visual elements and the beat signal output at the beat tempo to determine an entrainment level; The AA / AR modeling module further configures the processor to synchronize and dynamically adjust the AR visuals and the RAS output to the patient based on the determined entrainment level, where dynamically adjusting the AR visual content includes adjusting the defined spatial and temporal ordering of the visual elements based on the determined entrainment level. system.

2. 2. The system of claim 1, wherein the processor dynamically adjusts the AR visual content and the RAS based on the determined entrainment level by adjusting the beat tempo of the RAS to take into account the entrainment level, and adjusts the defined spatial and temporal ordering of the visual elements to be synchronized with the adjusted beat tempo.

3. Each of the beat signals is output at a respective output time, the tuning level is determined based on a timing of the repetitive motion relative to the respective output times of the beat signals. The system of claim 2.

4. The CTA module comprises: analyzing the time-stamped biomechanical data to identify each time of each of the repetitive movements; measuring a time relationship between said respective times of one or more of said repetitive movements and respective output times of one or more of said beat signals associated therewith; determining a synchronization margin based on the measured temporal relationship for one or more of the respective repetitive movements; configuring the processor to determine the tuning level by The processor is configured to dynamically adjust one or more of the output of the AR vision and the RAS based on the tuning room. The system of claim 3.

5. The CTA module further configures the processor to determine, based on one or more of physiological data and the biomechanical data measured for the patient, whether one or more of the biomechanical data or the physiological data satisfy training goal parameters; The AA / AR modeling module further configures the processor to dynamically adjust the AR visual content and the RAS output to the patient in response to the unachieved training goal parameter by adjusting the beat tempo of the RAS to account for the unachieved training goal parameter and by adjusting the defined spatial and temporal ordering of the visual elements to be synchronized with the adjusted beat tempo. The system of claim 2.

6. an AR video output device configured to present the AR visual content to the patient; an audio output device configured to output the RAS to the patient; the sensor associated with the patient and configured to measure the time-stamped biomechanical data of the patient, the sensor comprising an inertial measurement unit (IMU); The system of claim 1 further comprising:

7. a sensor associated with the patient and configured to measure physiological data of the patient, the training target parameter being a physiological parameter; Further equipped with The physiological parameters are one or more of heart rate, blood oxygenation, respiratory rate, VO2, electrical brain activity (EEG); The system of claim 5.

8. The AR visual content includes: a virtual treadmill, the virtual treadmill being animated such that a top surface of the virtual treadmill appears to be approaching the patient at a speed corresponding to the beat tempo; a plurality of footprints superimposed on the top surface of the virtual treadmill, the footprints being spatially arranged to appear to approach the patient at a speed corresponding to the beat tempo; an animated person performing the repetitive motion at a speed corresponding to the beat tempo; [0033] The visual scene includes one or more of: The system of claim 2.

9. 10. The system of claim 8, wherein modifying the AR visual content includes one or more of: altering the spacing of the footprints to account for changes in the beat tempo; altering the speed at which the animated person performs the repetitive motion; altering the speed at which a virtual top surface of a treadmill appears to move towards the patient; and altering the speed at which a virtual obstacle appears in front of the patient.

Images