Systems and methods for enhancing neurorehabilitation

Abstract

A method and system for augmented neurorehabilitation (ANR) for patients are disclosed. The ANR system generates rhythmic auditory stimuli (RAS) and visual augmented reality (AR) scenes, which are synchronized according to a common beat speed and output to the patient during a treatment session. Sensors worn by the patient capture biomechanical data related to repetitive movements performed by the patient in sync with the AR visual content and RAS. Critical thinking algorithms analyze the sensor data to determine the spatial and temporal relationships of the patient's movements relative to the visual and audio elements, and to determine the patient's entrainment level and progress toward clinical / therapeutic goals. Furthermore, a 3DAR modeling module configures the processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the determined entrainment level and whether training goals have been achieved.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to, U.S. Provisional Patent Application No. 63 / 054,599, filed July 21, 2020, entitled "Systems and Methods for Augmented Neurologic Rehabilitation" by McCarthy et al., and is further a partial continuation of, U.S. Patent Application No. 16 / 569,388, filed October 22, 2019, entitled "Systems and Methods for Neurologic Rehabilitation" by McCarthy et al., which is a continuation of, U.S. Patent Application No. 10,448,888, filed October 22, 2019, entitled "Systems and Methods for Neurologic Rehabilitation," which is based on and claims priority to, U.S. Patent Application No. 10,448,888, filed April 14, 2016, entitled "Systems and Methods for..." Priority is claimed in U.S. Provisional Patent Application No. 62 / 322,504, entitled “Neurologic Rehabilitation”. All of the above documents are incorporated herein by reference, as if they were described in their entirety herein. Technical Field

[0003] This disclosure generally relates to systems and methods for rehabilitating users with physical impairments by providing music therapy. Background Technology

[0004] Numerous controlled studies over the past decade have highlighted the clinical role of music in neurorehabilitation. For example, it is well known that regular music therapy can directly enhance cognition, motor function, and language. The process of listening to music enhances brain activity in multiple ways, stimulating bilateral networks in a wide range of brain regions associated with attention, semantic processing, memory, cognition, motor function, and emotional processing.

[0005] Clinical data support the claim that music therapy enhances memory, attention, executive function, and mood. PET scans of the neural mechanisms behind music have shown that pleasant music stimulates a wide network between cortical and subcortical regions, including the ventral striatum, nucleus accumbens, amygdala, insula, hippocampus, hypothalamus, ventral tegmentum, anterior cingulate cortex, orbitofrontal cortex, and ventromedial prefrontal cortex. The ventral tegmentum produces dopamine and has direct connections with the amygdala, hippocampus, anterior cingulate cortex, and prefrontal cortex. This mesocortical-limbic system, which can be activated by music, plays a crucial role in regulating arousal, mood, reward, memory, attention, and executive function.

[0006] Neuroscience research has revealed how the fundamental organizational processes of memory formation in music share a mechanism with non-musical memory processes. The basis of phrase grouping, hierarchical abstraction, and musical patterns bears direct resemblance to the temporal chunking principle of non-musical memory processes. This implies that music-activated memory processes can transform and enhance non-musical processes.

[0007] Therefore, improvements are still needed in equipment, systems, and methods to protect the use of user identities and securely provide personal information. Summary of the Invention

[0008] In one aspect of the disclosed subject matter, a system for enhanced neurorehabilitation of patients is provided. The system includes a computing system having a processor configured with software modules comprising machine-readable instructions stored in a non-transitory storage medium.

[0009] The software module includes an AA / AR modeling module, which, when executed by the processor, configures the processor to generate augmented-reality (AR) visual content and rhythmic auditory stimulation (RAS) for output to the patient during a therapy session. Specifically, the RAS includes a beat signal output at a beat rate, and the AR visual content includes visual elements moving in a prescribed spatial and temporal sequence based on the beat rate.

[0010] The system also includes an input interface for communicating with the processor to receive real-time patient data, including timestamped biomechanical data of the patient associated with repetitive movements performed in real time with AR visual content and RAS. Specifically, the biomechanical data is measured using patient-associated sensors.

[0011] The software module also includes a critical thinking algorithm (CTA) module, which configures the processor to analyze timestamped biomechanical data to determine the temporal relationship between the patient's repetitive movements and visual elements and beat signals output at a beat rate, in order to determine the entrainment level relative to target parameters. Furthermore, the AA / AR modeling module further configures the processor to synchronously and dynamically adjust to the patient's AR vision and RAS output based on the determined entrainment level.

[0012] According to another aspect, a method for enhanced neurorehabilitation for patients with physical injuries is provided. This method is implemented on a computer system having a physical processor configured with machine-readable instructions, which execute the method when executed.

[0013] The method includes the step of providing rhythmic auditory stimulation (RAS) to the patient via an audio output device during a therapeutic session. Specifically, the RAS includes a beat signal output at a beat tempo.

[0014] The method also includes the step of generating augmented reality (AR) visual content for output to the patient via an AR display device. Specifically, the AR visual content includes visual elements that move in a prescribed spatial and temporal sequence based on a beat speed and are output in sync with the RAS. The method also includes the step of instructing the patient to perform repetitive movements in a timely manner in accordance with the beat signal of the RAS and the corresponding movement of the visual elements of the AR visual content.

[0015] The method also includes the step of receiving real-time patient data, which includes timestamped biomechanical data of the patient related to repetitive movements performed by the patient in real time with AR visual content and RAS. Specifically, the biomechanical data is measured using sensors associated with the patient.

[0016] The method also includes the step of analyzing timestamped biomechanical data to determine the temporal relationship between the patient's repetitive movements and visual elements and the beat signal output according to the beat signal, in order to determine the entrainment potential. Furthermore, the method includes the steps of: synchronously and dynamically adjusting the AR visual content and RAS to output to the patient based on the determined entrainment potential that does not meet the prescribed entrainment potential, and continuing the treatment session using the adjusted AR visual content and RAS. Attached Figure Description

[0017] As illustrated in the accompanying drawings, the above and other objects, features, and advantages of the apparatus, systems, and methods described herein will become apparent from the following description of specific embodiments. The drawings are not necessarily drawn to scale, but are intended to emphasize the principles of the apparatus, systems, and methods described herein.

[0018] Figure 1 This is a diagram illustrating an exemplary embodiment of a system for treating a user by providing music therapy, according to the disclosed subject matter;

[0019] Figure 2 This is a diagram illustrating several components of a system for rehabilitating a user by providing music therapy, according to an exemplary embodiment of the disclosed subject matter;

[0020] Figure 3 This is a schematic diagram of a sensor for measuring the biomechanical movements of a patient, according to exemplary embodiments of the disclosed subject matter;

[0021] Figure 4 This is a diagram illustrating several components of a system according to exemplary embodiments of the disclosed subject matter;

[0022] Figure 5An exemplary display of components of a system for rehabilitating a user by providing music therapy, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0023] Figure 6 It is a flowchart of one implementation of the analysis process based on an exemplary embodiment of the disclosed subject matter;

[0024] Figures 7-10 This is a flowchart of one implementation of a process based on an exemplary embodiment of the disclosed subject matter;

[0025] Figure 11 It is a time-plotting diagram showing music and physical movements of a patient according to exemplary embodiments of the disclosed subject matter;

[0026] Figures 12-13 Patient responses according to exemplary embodiments of the disclosed subject matter are shown;

[0027] Figures 14-15 Patient responses according to exemplary embodiments of the disclosed subject matter are shown;

[0028] Figures 16-17 Patient responses according to exemplary embodiments of the disclosed subject matter are shown;

[0029] Figure 18 An implementation of a technique for gait training for a patient, according to exemplary embodiments of the disclosed subject matter, is shown;

[0030] Figure 19 An implementation of a technique for neglect training of patients, according to exemplary embodiments of the disclosed subject matter, is shown;

[0031] Figure 20 An implementation of a technique for patient intonation training according to an exemplary embodiment of the disclosed subject matter is shown;

[0032] Figure 21 An implementation of a technique for music stimulation training of a patient, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0033] Figure 22 An implementation of a technique for gross motor training of a patient, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0034] Figure 23 An implementation of a technique for grip strength training for a patient, according to exemplary embodiments of the disclosed subject matter, is shown;

[0035] Figure 24 An implementation of a technique for training voice prompts for patients, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0036] Figure 25An implementation of a technique for training minimally conscious patients according to exemplary embodiments of the disclosed subject matter is shown;

[0037] Figures 26-28 An implementation of a technique for patient attention training according to exemplary embodiments of the disclosed subject matter is shown;

[0038] Figure 29 An implementation of a technique for dexterity training for a patient, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0039] Figure 30 An implementation of a technique for oral motor training for a patient, according to an exemplary embodiment of the disclosed subject matter, is shown;

[0040] Figure 31 An implementation of a technique for respiratory training for a patient, according to exemplary embodiments of the disclosed subject matter, is shown;

[0041] Figure 32 This is a diagram illustrating an enhanced neuroologic rehabilitation, recovery or maintenance (ANR) system for providing treatment to a patient according to exemplary embodiments of the disclosed subject matter;

[0042] Figure 33 This is an example of the use of the disclosed subject matter. Figure 32 Graphical visualization of parameters, system response, and indicator / target parameters measured during a treatment session performed by the ANR system;

[0043] Figure 34 This describes exemplary embodiments and uses based on the disclosed subject matter. Figure 32 A graph illustrating exemplary results related to metabolic changes generated during a training session performed by the ANR system;

[0044] Figure 35 This is an exemplary augmented reality (AR) display generated by an ANR system for displaying to a patient during a treatment session, according to an exemplary embodiment of the disclosed subject matter;

[0045] Figure 36A This is an exemplary AR display generated by an ANR system for displaying to a patient during a treatment session, according to an exemplary embodiment of the disclosed subject matter;

[0046] Figure 36B This is an exemplary AR display generated by an ANR system for displaying to a patient during a treatment session, according to an exemplary embodiment of the disclosed subject matter;

[0047] Figure 37 An implementation of a technique for gait training by providing enhanced audio and visual stimulation to a patient, according to exemplary embodiments of the disclosed subject matter, is shown;

[0048] Figure 38 This is a conceptual illustration of a hybrid system and process diagram of an ANR system configured to implement gait training techniques, according to exemplary embodiments of the disclosed subject matter;

[0049] Figure 39 This is a conceptual illustration of exemplary embodiments based on the disclosed subject matter. Figure 38 A mixing system and process diagram of the enhanced audio (AA) device components of an ANR system;

[0050] Figure 40 This refers to an exemplary AR display generated by an ANR system for displaying to a patient during a treatment session, according to an exemplary embodiment of the disclosed subject matter; and

[0051] Figure 41 This is an exemplary AR display generated by an ANR system for displaying to a patient during a treatment session, based on an exemplary embodiment of the disclosed subject matter. Detailed Implementation

[0052] This invention generally relates to systems, methods, and apparatus for implementing a dynamic closed-loop rehabilitation platform system that monitors and guides changes in human behavior and function. These changes are manifested as shifts in language, movement, and cognition temporarily triggered by musical rhythm, harmony, melody, and dynamic cues.

[0053] In various embodiments of the present invention, the following are provided Figure 1 The illustrated dynamic closed-loop rehabilitation platform music therapy system 100 includes a sensor assembly and system 102, an edge processing assembly 104, a collector assembly 106, an analysis system 108, and a music therapy center 110. As will be described in more detail below, the sensor assembly, edge processing assembly, collector assembly, machine learning process, and music therapy center can be provided on various hardware components. For example, in one embodiment, the sensor assembly and edge processing assembly can be located or worn by the patient. In such an embodiment, the collector assembly and music therapy center can be provided on a handheld device. In such an embodiment, the analysis system can reside on a remote server.

[0054] Sensor system

[0055] Throughout this description, the term "patient" refers to an individual receiving music therapy treatment. The term "therapist" refers to an individual providing music therapy treatment. In some embodiments, the patient can interact with the system described herein to receive treatment in the absence of a therapist.

[0056] Sensor assembly 102 provides sensed biomechanical data about the patient. In some embodiments, the sensor assembly may include (1) a wearable wireless real-time motion sensing device or IMU (inertial measurement unit), (2) a wearable wireless real-time combined multi-zone plantar pressure / 6D motion capture (IMU) device, such as sensor 200, (3) a wearable wireless real-time electromyography (EMG) device, such as sensor 208, and (4) a real-time wireless near infrared (NIR) video capture device, such as imaging device 206 (see Figure 4 ).

[0057] like Figure 2 As shown, the systems and methods described herein are used to treat patients with gait disorders. Therefore, the exemplary sensor 200 can be a combined multi-zone plantar pressure / 6DOF motion capture device. While the patient walks during a music therapy session, sensor 200 records the patient's foot pressure and 6DOF motion distribution. In some embodiments, the foot pressure / 6DOF motion capture device has a variable recording duration interval, with a sampling rate of 100 Hz for a foot pressure distribution comprising 1 to 4 zones, resulting in 100 to 400 pressure data points per second per foot.

[0058] Sensor 200 may include a foot pressure pad 202 having a heel pad (for measuring one pressure zone, such as heel impact pressure) to a full insole (for measuring four pressure zones). Pressure measurements are performed by sensing changes in resistance in the transducer material caused by compression due to the patient's weight being transferred to the foot. These foot pressure maps are obtained for each sampling interval or at a specific moment during a music therapy session.

[0059] Sensor 200 may include a 6D motion capture device 204 that detects motion changes via a sensor based on a 6-DOF micro-electro-mechanical systems (MEMS), which determines linear acceleration A in three dimensions. x A y A z And rotational movements as pitch, yaw, and tilt. Sampling at 100Hz will generate 600 motion data points per second. These foot motion captures are obtained for each sampling interval or at specific moments during a music therapy session.

[0060] Multi-zone pressure sensing using a 6-DOF motion capture device allows for mappable spatial and temporal gait dynamics tracking during walking. Figure 3A schematic diagram of sensor 200 is shown in the figure.

[0061] From a systems perspective, such as Figure 4 As shown, patient P uses two foot sensors 200, one for each foot, designated right 200R and left 200L. In an exemplary embodiment, the right foot sensor 200R wirelessly transmits timestamp-internal measurement unit data and heel impact pressure data on a first channel (e.g., channel 5) in the IEEE 802.15.4 direct sequence spread spectrum (DSSS) RF band. The left foot sensor 200L wirelessly transmits timestamp-internal measurement unit data and heel impact pressure data on a second channel (e.g., channel 6) in the IEEE 802.15.4 direct sequence spread spectrum (DSSS) RF band. As described below, the therapist T may optionally use a tablet or laptop computer 220 including a wireless USB hub containing two IEEE 802.15.4 DSSS RF transceivers tuned to the first and second channels (e.g., channels 5 and 6) to capture the right / left foot sensor RF data. The handheld wireless trigger 250 is used to start and stop video and / or mark and index the time stream, as discussed in more detail below.

[0062] The video analytics domain can be used to extract patient semantic and event information about a treatment session. Patient actions and interactions are components that influence the treatment environment and the team during therapy. In some embodiments, one or more image capture devices 206 (e.g., video cameras) (see...) Figure 4 It is used in conjunction with time-synchronized video feeds. Any suitable video can be incorporated into the system to capture patient movement; however, near-infrared (NIR) video capture is useful for protecting patient privacy and reducing the amount of video data to be processed. The NIR video capture device captures NIR video images of the patient's body, such as the position of the patient's torso and limbs. Furthermore, it captures the patient's real-time dynamic gait characteristics based on a music therapy session. In some embodiments, the video is taken with a still camera, where the background is subtracted to segment the foreground pixels.

[0063] like Figure 4 As shown, when a therapy session begins, a tablet or laptop application triggers one or more video cameras 206. The therapist can stop or start the video cameras 206 via a handheld wireless trigger unit 250. This allows for the creation of tagged timestamp indexes in the captured biomechanical sensor data and video data stream.

[0064] In some embodiments, the patient may wear a wearable wireless real-time electromyography (EMG) device 208. The EMG sensor provides a distribution of activation of the major muscles involved in movement across the entire bipedal region. Such a sensor provides accurate data on the timing of muscle activation.

[0065] Edge processing

[0066] In some embodiments, edge processing is performed at sensor 200, where sensor data is captured from the IMU and pressure sensor. This sensor data is filtered and grouped into various array sizes for further processing into frames reflecting extracted attributes and features, and these frames are transmitted, for example wirelessly, to collector 106 on a tablet or laptop. It should be understood that the raw biomechanical sensor data obtained from sensor 200 may alternatively be transmitted to a remote processor for collection by edge processing functions.

[0067] Wearable sensors 200 and 208 and video capture device 206 generate sensor data streams, which are processed holistically to facilitate biomechanical feature extraction and classification. Sensor fusion, which combines the outputs of multiple sensors capturing common events, captures results better than any single component sensor input.

[0068] By capturing patient activity within a music therapy setting, this approach formalizes interactions applicable to music therapy and develops patient-specific and generalized formal indicators of music therapy performance and efficacy. Extracting and analyzing video features allows for the capture of semantic, high-level information about patient behavior.

[0069] When processing the video, a learned background subtraction technique is used to create a background model that incorporates any variations in lighting conditions and occlusion in the physical area where the music therapy takes place. The result of background subtraction is a binary foreground map of a foreground blob array with a two-dimensional contour. Therefore, the video is segmented into individual image frames for future image processing and sensor fusion. Additional metadata is provided to the video information by incorporating edge-processed sensor data from the IMU, foot pressure pad, and EMG sensors. Sensor data can be time-synchronized with other data using an RF trigger. Data can be sent directly to the collector, stored in the internal board's memory, or analyzed at the edge while running OpenCV libraries.

[0070] The edge processor 104 may be a microprocessor, such as a 32-bit microprocessor, which is integrated into a foot pressure / 6 degrees of freedom motion capture device that is capable of performing fast multi-zone scans at a rate of 100 to 400 complete foot pressure / 6 degrees of freedom motion distributions per second.

[0071] The foot pressure / 6-DOF motion capture device collects foot pressure / 6-DOF motion distribution data for real-time gait analysis, thereby performing feature extraction and classification. In some embodiments, the foot pressure / 6-DOF motion capture device initializes the microcontroller unit (MCU), continuous operator process (COP), general purpose input / output (GPIO), serial peripheral interface (SPI), and interrupt request (IRQ), and sets the desired RF transceiver clock frequency by calling routines including MCUInit, GPIOInit, SPIInit, IRQInit, IRQACK, SPIDrvRead, and IRQPinEnable. MCUInit is the main initialization routine, which disables the MCU watchdog and sets the timer module to use the bus clock (BUSCLK) as a reference, prescaled to 32.

[0072] The status variable gu8RTxMode is set to SYSTEM_RESET_MODE, and the routines GPIOInit, SPIInit, and IRQInit are called. The status variable gu8RTxMode is set to RF_TRANSCEIVER_RESET_MODE, and the IRQFLAG is checked to see if the IRQ is asserted. First, the RF transceiver interrupt is cleared using SPIDrvRead, and then the RF transceiver's ATTNIRQ interrupt is checked. Finally, for MCUInit, PLMEPyReset is called to reset the physical MAC layer, IRQACK (acknowledging pending IRQ interrupts), and IRQPinEnable, which pins Enable, IE, and IRQCLR to the negative edge of the signal.

[0073] The foot pressure / 6DOF motion sensor 200 will wait for a response from the foot pressure / 6DOF motion acquisition node, for example, 250 milliseconds, to determine whether to perform a default full foot pressure scan or to initiate a mapped foot pressure scan. In the case of a mapped foot pressure scan, the foot pressure / 6DOF motion acquisition node will send foot pressure scan mapping configuration data to the appropriate electrodes.

[0074] One aspect of the analysis pipeline is the feature set engineering process, which defines the captured sensor values ​​and the resulting sensor fusion values. These values ​​are used to create feature vectors to define the input data structure for analysis. The representative values ​​are Ax(i), Ay(i), Az(i), and Ph(i); where i is the i-th sample; Ax(i) is the acceleration in the x-direction, which is lateral relative to the foot sensor; Ay(i) is the acceleration in the y-direction, which is forward relative to the foot sensor; Az(i) is the acceleration in the z-direction, which is upward relative to the foot sensor; and Ph(i) is the heel impact pressure. The sensor values ​​are shown in Table 1.

[0075]

[0076] In some embodiments, sensor fusion techniques use heel impact pressure values ​​Ph(i) to “gated” the analysis of the following exemplary feature values ​​to derive a data window, as described below. For example, “start” (beginning) may be determined based on heel pressure exceeding a threshold indicating heel impact, while “stop” may be determined based on heel pressure falling below a threshold indicating heel lift-off, as shown in Table 2 below. It is understood that heel impact pressure is one example of a parameter that can be used for “gated” analysis. In some embodiments, “gated” is determined using IMU sensor data, video data, and / or EMG data.

[0077]

[0078]

[0079] Calculate higher-level feature values ​​based on the fused sensor values, such as the example values ​​in Table 3:

[0080]

[0081]

[0082] The system described in this article can "gated" patient biomechanical data or provide a "window" for patient biomechanical data. Gating biomechanical data is very useful for repetitive patient movements, such as repetitive steps during a patient's walk. Sensor data from one or more sources, such as pressure sensors, IMU sensors, video data, and EMG data, are used to identify repetitive motion cycles over time. For example, when a patient walks, foot pressure repeatedly increases and decreases as the patient's foot contacts and is lifted off the ground. Similarly, the foot's velocity increases as it moves forward and decreases to zero when it lands on the ground. As a further example, the Y-position or height of the patient's foot cycles between a low position (on the ground) and a high position (approximately midway through a step). The "gating" technique identifies repetitive cycles or "windows" in this data. For a patient walking, this cycle is repeated with each step. Although there may be differences between cycles (e.g., between steps), certain patterns repeat with each cycle. Selecting the start time (start time) for each cycle involves locating identifiable points (maximum or minimum values) of biomechanical parameters. The parameter selection for the start time is based on available data. Therefore, in some embodiments, the moment when the heel impact pressure exceeds a threshold can be used to calibrate the start time of each cycle. (See, for example) Figure 5 Pressures 316a and 316b include cyclic characteristics. "Start" can be determined when the pressure exceeds a threshold. Similarly, the start time can be calibrated when the foot's speed drops to zero.

[0083] In some embodiments, raw frame data is preprocessed to acquire real-time data and "gated," for example, by identifying a window and then analyzing the data within that window to identify outliers and perform data analysis, such as exponential analysis, averaging the data across multiple windows. The fusion of sensor data (by including IMU data and heel impact pressure data) allows for more precise identification of the onset time of individual strides or other repetitive motion units compared to data from a single sensor. Sensor data captured within a single stride is considered a "window," and information extracted from this analysis includes, for example, stride length, step count, gait, stride occurrence time, distance traveled, stance / swing phase, double support time, velocity, symmetry analysis (e.g., between the left and right legs), outward swing, dragging, power vector, lateral acceleration, stride width, variability in each of these dimensions, additional parameters derived from the above information, etc. Feature extraction can be processed on a microprocessor chip (e.g., a 32-bit chip). Wireless synchronous gating of biomechanical sensor data capture and video data capture capabilities allow for the creation of time-series templates.

[0084] Patients or therapists can index data during music therapy sessions. The aforementioned "gating" function can be used to associate abnormalities with specific strides or steps. For example, a therapist can observe specific abnormalities or behaviors (e.g., anomalies or events) in a patient's movement. The indexing function allows therapists to access data via a user interface on a handheld tablet or laptop (e.g., ...). Figure 4 The wireless trigger unit 250 shown can be used or voice control can be used to initiate (e.g., capture) the “recording” of abnormal situations or behaviors. A notation system containing timestamps and annotations can be created, such as a patient “tripping” while walking. This indexing helps in creating time series templates. These time series templates will be studied to examine treatment session events and develop time series templates for training machine learning algorithms such as non-linear multi-layered perceptrons (NLMLP), convolutional neural networks (CNN), and recurrent neural networks (RNN) with long short-term memory (LSTM).

[0085] In one embodiment, a communication protocol is provided to transmit sensor data from edge processing 104 (e.g., at sensor 200) to collector 106. See Table 4 below. In some embodiments, if the connection is idle for more than 100 ms, the RF has timed out.

[0086]

[0087] In one embodiment, a foot pressure sensor region scan is performed via the FootScan routine, where the FootDataBufferIndex is initialized and the foot pressure sensor region is activated by enabling MCU orientation mode for output [PTCDD_PTCDDN = output] and setting the relevant port line low [PTCDDPTCD6 = 0]. When the foot pressure sensor region is activated based on the foot pressure sensor region scan map, the foot pressure sensor region connected to the MCU analog signal port is sampled, and then the current and voltage readings are converted into digital form (i.e., time zone foot pressure).

[0088] Several variables (such as FootDataBufferIndex and IMUBufferIndex) are used to prepare the IEEE 802.15.4 RF packet gsTxPacket.gau8TxDataBuffer[], which is used to send the data to be used in FootDataBuffer[] and IMUBuffer[]. The RF packet is sent using the RFSendRequest(&gsTxPacket) routine. This routine checks to see if gu8RTxMode is set to IDLE_MODE and uses gsTxPacket as a pointer to call the RAMDrvWriteTx routine, which then calls SPIDrvRead to read the contents of the RF transceiver's TX packet length register. Using these contents, the mask length setting is updated, then the CRC is incremented by 2, and the code byte is incremented by 2.

[0089] The SPISendChar call sends 0x7E bytes, the second code byte, followed by a call to SPIWaitTransferDone to verify completion. After sending these code bytes, a for loop sends the remainder of the data packet, where psTxPkt→u8DataLength+1 is the iteration count of the SPISendChar, SPIWaitTransferDone, and SPIClearRecieveDataReg sequences. Once complete, the RF transceiver loads the data packet to be sent. ANTENNA_SWITCH is set to transmit, LNA_ON mode is enabled, and finally, an RTXENAssert call is made to actually send the data packet.

[0090] Collector

[0091] The primary functions of collector 106 are to capture data from edge processing 104, transmit data to and receive processed data from analysis system 108, and transmit data to music therapy center 110, as described below. In some embodiments, collector 106 provides control functions, such as a user interface for logging in, configuring the system, and interacting with the user, and includes a display unit for visualizing / displaying data. Collector 106 may include lightweight analytical or machine learning algorithms for classification (e.g., lateral tremor, asymmetry, instability, etc.).

[0092] Collector 106 receives body, motion, and positioning data from edge processor 104. The data received at collector 106 may be raw or processed at edge 104 before being transmitted to the collector. For example, collector 106 receives fused sensor data that has been “windowed” and feature extracted. The transmitted data may include two levels of data: (1) RF packets transmitted from the right / left foot sensors as described in Table 1, and (2) RF packets containing higher-level attributes and features transmitted from the left / right foot sensors as described in Tables 2 and 3. Collector 106 stores the data locally. In some embodiments, collector 106 classifies motion based on the received data, for example, by comparing it to a locally stored (pre-downloaded from an analysis system) or sent to the analysis system for classification. The collector may include a display unit to visualize / display the data.

[0093] In some embodiments, collector 106 operates on a local computer including memory, a processor, and a display. Exemplary devices with the collector installed may include augmented reality (AR) devices, virtual reality (VR) devices, tablets, mobile devices, laptops, desktop computers, etc. Figure 2 A handheld device 220 with a display 222 and performing collector functions is shown. In some embodiments, connection parameters for transmitting data between the patient sensor and the collector include the use of Device Manager in Windows (e.g., baud rate: 38400, data bits: 8; parity: none, stop bits: 1). In some embodiments, the collector 106 includes a processor held or worn by the music therapy patient. In some embodiments, the collector 106 includes a processor that is remote from the music therapy patient and carried by the therapist, and is wirelessly or via a wired connection to the music therapy patient.

[0094] In one embodiment, a foot pressure / 6DOF motion collection node captures RF transmission packets containing real-time foot pressure / 6DOF motion distribution data from a foot pressure / 6DOF motion capture device. This is initiated by the foot pressure / 6DOF motion collection node, which creates an RF packet receive queue driven by a callback function when RF transceiver packet reception is interrupted.

[0095] When an RF data packet is received from the foot pressure / 6-DOF motion capture device 200, it is first checked to determine whether it is from a new foot pressure / 6-DOF motion capture device or an existing one. If it is from an existing foot pressure / 6-DOF motion capture device, the RF data packet sequence number is checked to determine continuous synchronization before further data packet analysis. If it is from a foot pressure / 6-DOF motion capture device, a foot pressure / 6-DOF motion capture device environment state block is created and initialized. The environment state block includes information such as foot pressure distribution.

[0096] Above the RF packet session-level process used for node-to-node communication is the analysis of the RF packet data payload. This payload contains the foot pressure distribution based on the current variable pressure after 6 degrees of freedom actions. Its structure is as follows: |0×10|Start|F1|F2|F3|F4|Ax|Ay|Az|Pi|Yi|Ri|XOR checksum|.

[0097] The IEEE 802.15.4 standard specifies a maximum packet size of 127 bytes. The Time Synchronized Mesh Protocol (TSMP) reserves 47 bytes for operation, leaving 80 bytes for the payload. IEEE 802.15.4 is compliant with 2.4 GHz Industrial, Scientific, and Medical (ISM) band radio frequency (RF) transceivers.

[0098] The RF module contains a complete 802.15.4 physical layer (PHY) modem designed to support the IEEE 802.15.4 wireless standard in peer-to-peer, star, and mesh networks. It works in conjunction with the MCU to create the necessary wireless RF data links and networks. The IEEE 802.15.4 transceiver supports 250kbps O-QPSK data in a 5.0MHz channel, along with full spread spectrum encoding and decoding.

[0099] In some embodiments, control, status reading, data writing, and data reading are performed through the RF transceiver interface port of the sensing system node device. The sensing system node device's MPU accesses the RF transceiver of the sensing system node device via interface "transactions," in which multiple bursts of byte-length data are transmitted on the interface bus. Depending on the transaction type, each transaction has three or more bursts. Transactions always involve either a read or write access to a register address. The length of the data associated with any single register access is always 16 bits.

[0100] In some embodiments, the control of the RF transceiver and data transmission of the foot pressure / 6DOF motion collector node is implemented through a Serial Peripheral Interface (SPI). While the normal SPI protocol is based on 8-bit transmission, the RF transceiver of the foot pressure / 6DOF motion collector node applies a higher-level transaction protocol based on multiple 8-bit transmissions per transaction. A single SPI read or write transaction consists of an 8-bit header transmission and two 8-bit data transmissions.

[0101] The header indicates the access type and register address. The following bytes represent the read or write data. SPI also supports recursive "data burst" transactions, in which additional data transfers may occur. Recursive mode is primarily used for packet RAM access and rapid configuration of the foot pressure / 6DOF motion collection node RF.

[0102] In some embodiments, all foot pressure sensor areas are scanned sequentially, and the entire process is repeated until a reset condition or inactive power-off mode is reached. The 6-DOF motion is captured via a serial UART interface between the MCU and the inertial measurement unit (IMU). The sampling rate for all sensing dimensions (i.e., Ax, Ay, Az, pitch, yaw, and roll) is 100–300 Hz, and the sampled data is stored in the IMUBuffer[].

[0103] Call SPIDryWrite to update the TX packet length field. Next, call SPICEClearRecieveStatReg to clear the status register, and then call SPICEClearRecieveDataReg to clear the receive data register, preparing the SPI interface for reading or writing. After the SPI interface is ready, call SPISendChar to send a 0xFF character (representing the first code byte), and then call SPIWaitTransferDone to verify that the transmission was complete.

[0104] Figure 5 An exemplary output 300 may be provided on the display 222 of a handheld device. For example, when providing treatment for a patient's gait, the display output 300 may include a portion of the right foot 302 and a portion of the left foot 304. Over time, the display of the right foot includes accelerations Ax310a, Ay312a, and Az314a, as well as foot pressure 316a. Similarly, the display of the left foot includes accelerations Ax310a, Ay312a, and Az314a, as well as foot pressure 316a.

[0105] Classification is understood as the correlation between data (e.g., sensor fusion data, feature data, or attribute data) and real-world events (e.g., patient activities or treatment). Typically, classification is created and performed on the analysis system 108. In some embodiments, the collector 106 has local copies of some "templates" so that the input sensor data and feature-extracted data can be classified at the collector or analysis system.

[0106] The environment refers to the circumstances or facts that constitute the setting of an event, statement, situation, or idea. Environmental awareness algorithms examine the "who," "what," "when," and "where" in relation to the environment and time in which the algorithm is executed for specific data. Some environmental awareness actions include identity, location, time, and ongoing activities. When using environmental information to formulate deterministic actions, the environmental interface occurs between the patient, the environment, and the music therapy session.

[0107] The patient's response environment to a music therapy session may involve an algorithmic layer that interprets fused sensor data to infer higher-level information. These algorithms extract the patient's response environment. For example, analyzing the patient's biomechanical gait sequence as it relates to a specific part of the music therapy session. In one example, "lateral tremor" is the classifier of interest. Thus, in cases of less lateral tremor, it is determined that the patient's gait is more fluid.

[0108] Analysis System

[0109] Analysis system 108 (sometimes referred to as a backend system) stores large modules / archives and includes machine learning / analysis processing, as well as the modules described herein. In some embodiments, a web interface is provided for logging in to view archived data, and a dashboard is also provided. In some embodiments, analysis system 108 resides on a remote server computer that receives data from collector 106 running on a handheld unit 220, such as a handheld device or tablet. It is conceivable that the processing power required to perform the analysis and machine learning functions of analysis system 108 may also reside on the handheld device 220.

[0110] Data is transmitted from collector 106 to analysis system 108 for analysis and processing. For example... Figure 6As shown, the analysis process 400 includes a user interface 402 for receiving data from the collector 106. A database storage 404 receives input data from the collector 106 for storage. Training data, as well as the output of the analysis process (e.g., the integrated machine learning system 410), can also be stored on the storage 404 to facilitate the creation and refinement of predictive models and classifiers. A data bus 406 allows data to flow through the analysis process. A training process 408 is performed on the training data to derive one or more predictive models. The integrated machine learning system 410 utilizes the predictive models. The output of the integrated machine learning system 410 is an aggregation of these predictive models. This aggregated output is also used for the classification requirements of the template classifier 412, such as tremor, symmetry, mobility, or learned biomechanical parameters, such as entrainment, initiation, etc. An API 418 connects to the collector and / or the music therapy center. The therapy algorithm 414 and the predictive algorithm 416 include multilayer perceptron neural networks, hidden Markov models, radial basis function networks, Bayesian inference models, etc.

[0111] An exemplary application of the system and method described in this paper is the analysis of patient biomechanical gait. Gait sequences are feature-extracted into a series of typical features. The presence of these and other features in the captured sensor fusion data informs an environmental detection algorithm that the patient's biomechanical gait sequence is valid. Biomechanical gait sequence capture requires robust environmental detection, followed by abstraction of a representative population of music therapy patients.

[0112] One example of this type of activity is observing a patient's location at a given point in time and their response to music therapy at that time. Identifying and correlating patient responses to music therapy allows for the identification of specific patterns in patient responses. Then, by creating a baseline of patient responses and correlating it with future patient responses, specific music therapy protocols can be benchmarked and their performance and efficiency analyzed.

[0113] By combining motion sensing with distance metrics captured using gait biomechanics, patient path trajectories are determined by leveraging temporal and spatial variations / deviations between two or more music therapy sessions. Features are extracted and categorized from this sensor fusion data capture to label various key patient treatment responses. Further sensor fusion data analysis uses histograms to allow for the detection of initial music therapy response patterns.

[0114] For the analysis of sensor fusion data from music therapy sessions, initially, a patient-specific Bayesian inference model using Markov chains was employed. The states of the chain represent patient-specific response patterns captured from baseline music therapy sessions. Inference is based on knowledge of the occurrence of patient response patterns at each sample interval and their temporal relationship with previous states.

[0115] The prediction routine, a Multilayer Perceptron Neural Network (MLPNN), uses a directed graph-based model with a top-level root node that predicts the need to reach subsequent nodes and obtain the sensor fusion data feature vector of the patient. This sensor fusion data feature vector contains motion data for time-series processing, music signature data, and video image data that is particularly important for further processing. In this case, the directed graph looks like a tree drawn upside down, with the leaves at the bottom and the root node as the root. From each node, the routine can move to the left (the leftmost node in the next layer below the root node) and select the left child node as the next observation node, or it can move to the right (the rightmost node in the next layer below the root node), based on the value of a variable whose index is stored in the observation node. If the value is less than a threshold, the routine moves to the left node; if it is greater than the threshold, the routine moves to the right node. These regions (here, the left and right) constitute the prediction space.

[0116] The model uses two types of input variables: ordinal variables and categorical variables. Ordinal variables are values ​​compared to thresholds stored in nodes. Categorical variables are discrete values ​​that are tested to determine whether they belong to a finite subset of values ​​and are stored in nodes. This can be applied to various categories. For example, mild, moderate, and severe can be used to describe tremor and are examples of categorical variables. Conversely, a fine-grained range of values ​​or numerical scale can be used to describe tremor similarly, but in a numerical manner.

[0117] If the categorical variable belongs to a finite set of values, the routine moves to the left node; otherwise, it moves to the right node. Within each node, a pair of entities—variable_index and decision_orule (threshold / subset)—is used for decision-making. This pair is called the split, and the split variable is variable_index.

[0118] Once a node is reached, the value assigned to that node is used as the output of the prediction routine. The multilayer perceptron neural network is recursively built starting from the root node. As previously mentioned, all training data, feature vectors, and responses are used to split the root node; where the entity: variable_index, decision_le (threshold / subset) segments the predicted region. Within each node, the optimal decision rule for the best master split is found based on the Gini "purity" criterion used for classification and the sum of squared errors used for regression. The Gini index is based on a measure of the total variance of the set of classes. The Gini "purity" criterion refers to a small Gini index value indicating that the node primarily contains observations from a single class, which is ideal.

[0119] Once the multilayer perceptron neural network is built, it can be pruned using a cross-validation routine. To avoid overfitting, some branches of the tree are pruned. This routine can be applied to independent decision-making. As mentioned above, a significant property of the decision-making algorithm (MLPNN) is its ability to compute the relative decision power and importance of each variable.

[0120] Variable importance rating is used to determine the most frequent interaction types in the patient interaction feature vector. Pattern recognition begins by defining a decision space suitable for distinguishing different categories of music therapy responses and events. The decision space can be represented by a graph with N dimensions, where N is the number of attributes or measures considered to represent music therapy responses and events. The N attributes form a feature vector, or signature, which can be plotted in the graph. After a sufficient number of samples are input, the decision space reveals clusters of music therapy responses and events belonging to different categories, which are used to associate new vectors with these clusters.

[0121] The dynamic closed-loop rehabilitation platform music therapy system utilizes multiple deep learning neural networks to learn and recall patterns. In one embodiment, an adaptive radial basis function (RBF) model generator is used to construct a nonlinear decision space. New vectors can be computed using the RBF model and / or a K-nearest neighbor classifier. Figure 6 The workflow of the machine learning subsystem of the music therapy system in the dynamic closed-loop rehabilitation platform is shown.

[0122] Figure 7 A supervised training process 408 is illustrated, comprising multiple training samples 502. For example, the input would be features as described in Table 3 above, and the example output would be items such as tremor, asymmetry, and power, the degree of these items, predictions of variations, and classifications of patient recovery. It is understood that new outputs are learned as part of this process. This provides a foundation for higher-level abstractions in prediction and classification, as it is applicable to various use cases (e.g., different disease states, combinations with medications, notifications to providers, fitness, and fall prevention). These training samples 502 are run using learning algorithms A1504a, A2504b, A3504c…AN504n to obtain predictive models in M1506a, M2506b, M3506c…MN506n. Exemplary algorithms include multilayer perceptron neural networks, hidden Markov models, radial basis function networks, and Bayesian inference models.

[0123] Figure 8An integrated machine learning system 410 is illustrated as an aggregation of prediction models M1506a, M2506b, M3506c…MN506n on sample data 602 (e.g., feature extraction data) to provide multiple prediction result data 606a, 606b, 606b…606n. Given multiple prediction models, an aggregation layer 608 (e.g., including decision rules and voting) is used to derive output 610.

[0124] The MRConvNet system has two layers. The first layer is a convolutional layer with average pooling support. The second layer is a fully connected layer that supports multivariate logistic regression. Multivariate logistic regression, also known as Softmax, is a generalization of logistic regression that handles multiple classes. In the case of logistic regression, the labels are binary.

[0125] Softmax is a model used to predict the probabilities of different possible outputs. The following assumes that the final output layer of Softmax is a multi-class classifier with m discrete classes:

[0126] Y1=Softmax(W11*X1+W12*X2+W13*X3+B1) [1]

[0127] Y2=Softmax(W21*X1+W22*X2+W23*X3+B2) [2]

[0128] Y3=Softmax(W31*X1+W32*X2+W33*X3+B3) [3]

[0129] Ym=Softmax(Wm1*X1+Wm2*X2+Wm3*X3+Bm) [4]

[0130] Overall: Y = softmax(W*X + B) [5]

[0131] Softmax(X)i=exp(Xi) / Sumofexp(Xj), j=1 to N [6]

[0132] Where Y = classifier output; X = sample input (all scaled (normalized) feature values); W = weight matrix. For example, classification would score asymmetry, such as "moderate asymmetry score of 6 out of 10 (10 points for high asymmetry, 0 points for no asymmetry)" or gait mobility "gait mobility score of 8 out of 10 for normal," etc. Analysis pipelines include... Figure 9 As shown.

[0133] Softmax regression allows handling multiple classes with more than two elements. For logistic regression: P(x) = 1 / (1 + exp(-Wx)), where W contains the model parameters trained to minimize the cost function. Furthermore, x is the input feature vector, and...

[0134] ((x(1),y(1)),…,(x(i),y(i))) [7]

[0135] Let represent the training set. For multi-class classification, Softmax regression is used, where y can take N distinct values ​​representing the class, instead of 1 and 0 in the binary case. Therefore, for the training set ((x(1),y(1)),…,(x(i),y(i))), y(n) can be any value in the range of class 1 to N.

[0136] Next, p(y = N|x; W) is the probability of each value i = 1, ..., N. The following is an arithmetic explanation of the Softmax regression process:

[0137] Y(x)=(p(y=1|x;W),p(y=2|x;W),…p(y=N|x;W)) [8]

[0138] Where Y(x) is the hypothetical answer, given input x, outputting the probability distribution of all classes such that their normalized sum is 1.

[0139] The MR ConvNet system takes each windowed biomechanical data frame as a vector and convolves it with each biomechanical template filter as a vector. It then generates a response using an average pooling function that averages the feature responses. The convolution process computes Wx while adding any biases, and then passes it to a logistic regression (S-shaped) function.

[0140] Next, in the second layer of the MR ConvNet system, the subsampled biomechanical template filter responses are shifted into a two-dimensional matrix, where each column represents a windowed biomechanical data frame as a vector. The Softmax regression activation process is now initiated using the following method:

[0141] Y(x)=(1 / (exp(Wx)+exp(Wx)+….+exp(Wx))*(exp(Wx),exp(Wx),…,(exp(Wx))[9]

[0142] The MRConvNet system is trained using an optimization algorithm (gradient descent), which defines and minimizes a cost function J(W):

[0143] J(W)=1 / j*((H(t(j=1),p(y=1|x;W)+H(t(j=2),p(y=2|x;W)+…+H(t(j),p(y=N|x;W))

[10]

[0144] Where t(j) is the target class. This averages the cross-entropy of all j training samples. The cross-entropy function is:

[0145] H(t(j),p(y=N|x;W)=-t(j=1)*log(p(y=1|x;W))+t(j=2)*log(p(y=2|x;W))+…+t(j)*p(y=N|x;W)

[11]

[0146] exist Figure 10 In this ensemble machine learning system 408, multiple predictive models are included, such as template series 1 (tremor) 706a, template series 2 (symmetry) 706b, template series 3 (fluidity) 706c…, which are applied to the conditional input 702 as additional templates (other learned biomechanical parameters, such as entrainment, initiation, etc.) 706n. For example, these could be: stride lengths (x1, x2) of right and left features, variances of stride lengths (x3, x4) of right and left features, pace (x6, x7) of right and left features, variances of pace (x8, x9) of right and left features, etc., where the samples (x1, x2, ..., xn) are referred to as vector X, which is input into the ML algorithm set 702. These are conditionally referenced, normalized, and / or scaled inputs. The aggregate classifier 708 outputs information such as tremor scale, symmetry scale, fluidity scale, etc.

[0147] Music Therapy Center

[0148] Music therapy center 110 is in the processor (such as Figure 2 The decision-making system runs on a handheld device or laptop computer (220). The music therapy center 110 takes input from features in the extracted sensor data at collector 106, compares them with the definition process used to deliver therapy, and then delivers the content of the auditory stimulation played through the music delivery system 230.

[0149] Embodiments of the present invention use environmental information to determine the cause of a situation, and then encode the observed actions, which leads to dynamic and modulated changes in the system state, thereby enabling music therapy sessions in a closed-loop manner.

[0150] The interaction between the patient and the music therapy session provides real-time data for determining the patient's environmental awareness, including movements, postures, strides, and gait responses. After the input data is collected at the sensing node (at the sensor), the embedded node processes the environmental awareness data (at the edge processing) and provides immediate dynamic actions and / or transmits the data to the analysis system 108 (e.g., a processing cloud environment based on a resilient network) for storage and further processing and analysis.

[0151] Based on input, the program will take any existing song content and modify the tempo, major / minor chords, beat, and musical cues (e.g., melody, harmony, rhythm, and dynamics cues). The system can overlay a metronome onto existing songs. The song content can be beat-mapped (e.g., if W responds to an AV or MP3 file) or in MIDI format, allowing precise knowledge of the beat's timing to calculate the entrained potential. Sensors on the patient can be configured to provide tactile / vibrational feedback pulses at the points of musical content.

[0152] Example

[0153] This article describes an exemplary application of the method. Gait training analysis establishes a real-time relationship between the musical beats played for the patient and the individual steps taken by the patient in response to those specific musical beats. As described above, gating analysis is used to determine a data window that varies with each step or repetitive movement. In some embodiments, the start of the window is determined as the time when the heel impact pressure exceeds a threshold (or other sensor parameter). Figure 11 This is an exemplary time plot showing the musical beats, "time beats," and the steps taken by the patient, "time steps." Therefore, the start time in this case is related to the "time step." Specifically, the plot shows the musical time beat 1101 at time beat 1. After a period of time, the patient takes a step at time step 1 in response to time beat 1001, i.e., time step 1102. The entrained potential 1103 represents the delay (if any) between time beat 1 and time step 1.

[0154] Figures 12-13 An example of using the system described herein to clip a patient's gait is shown. Figure 12 The diagram illustrates a "perfect" entrainment, such as a constant entrainment potential of zero. This occurs when there is no delay or a negligible delay between the time tick and the corresponding time step taken in response to the time tick. Figure 13 This illustrates phase shift entrainment, such as a non-zero entrainment potential that remains constant or has minimal change over time. This occurs when there is a consistent delay within tolerance between the time tick and the time step.

[0155] Continue to refer to Figure 11The EP ratio is calculated as the ratio of the duration between time beats to the duration between time steps:

[0156]

[0157] Wherein, time beat 11101 corresponds to the time of the first musical beat, and time step 11102 corresponds to the time of the patient's step in response to time beat 1. Time beat 21106 corresponds to the time of the second musical beat, and time step 21108 corresponds to the time of the patient's step in response to time beat 2. The goal is EP ratio = 1 or EP ratio / coefficient = 1. The coefficient is determined as follows:

[0158]

[0159] This coefficient allows for the subdivision of the beat, or for someone to step on 3 out of every 3 or 4 beats. It provides flexibility for different scenarios.

[0160] Figure 14 and 15 The entrainment response of patients using the technique described herein over time is shown. Figure 14 (Left Y-axis: EP ratio; Right Y-axis: beats per minute; X-axis: time) The graph shows the scattering at point 1402, which represents the average EP ratio of the first patient's gait. The graph shows an upper limit of +0.1 at 1404 and a lower limit of -0.1 at 1406. Line 1408 shows the velocity over time (starting at 60 beats per minute, increasing to 100 bpm). Figure 14 It shows that when the speed increases from 60 bpm to 100 bpm, the EP ratio remains around 1 (±0.1). Figure 15 The EP ratio of the second patient's gait is shown, where the EP ratio also remained close to 1 (±0.1) as the speed increased from 60 bpm to 100 bpm.

[0161] Figure 16 and 17 (Y-axis: clamping potential, X-axis: time) shows the responses of two patients to changes in temporal beat (e.g., speed) and / or chord changes, changes in tactile feedback, changes in foot cues (e.g., left-right or left-right cane cues), etc. Figure 16 The diagram shows a time-based plot where the patient's gait is balanced with either "perfect entrainment" (a constant zero or negligible entrainment potential) or a constant phase-shifted entrainment potential. As shown, a certain time period, known as the golden time 1602, is required for balance to occur. Figure 17The diagram illustrates time-based plotting where the patient's gait is unbalanced, for example, failing to achieve perfect entrainment or a constant phase-shift entrainment potential after a change in tempo. The golden time is useful because it represents a dataset independent of the accuracy of entrainment measurement. The golden time parameter can also be used to screen the suitability of future music. For example, music that a patient exhibits a longer golden time value when using is less suitable for treatment.

[0162] Figure 18 The technique used for gait training is illustrated, where repetitive movements refer to the steps a patient takes while walking. Gait training is suitable for individual patient groups, diagnoses, and conditions to provide personalized and individualized musical interventions. Based on input, the program modifies the content, pacing, major / minor chords, gauge, and musical cues (e.g., melody, harmony, and dynamics cues) where applicable. The program can provide a passive music playlist for regular use by selecting music using birth date, listed musical preferences, and entrainment tempo. Key inputs to gait training are the user's pacing, symmetry, and stride length when performing physical activities such as walking. The program provides tactile / vibrational feedback at the music's BPM using connected hardware. Suitable populations for gait training include patients with traumatic brain injury (TBI), stroke, Parkinson's disease, MS, and the elderly.

[0163] The method begins at step 1802. In step 1804, biomechanical data is received at collector 106 based on data from sensors (e.g., sensors 200, 206, 208). The biomechanical data includes data on initiation, stride length, gait, symmetry, assistive devices, or other such patient characteristics stored and generated by analysis system 108. Exemplary biomechanical data parameters are listed in Tables 1, 2, and 3 above. Baseline conditions are determined by one or more data sources. First, the patient's gait is sensed without any music being played. Sensor and characteristic data regarding the patient's initiation, stride length, gait, symmetry, assistive devices, etc., include patient baseline biomechanical data for the treatment session. Second, sensor data from previous sessions of the same patient, as well as any higher-level classification data from analysis system 108, include the patient's historical data. Third, sensor data and higher-level classification data from patients in other similar locations include population data. Therefore, baseline conditions may include data from one or more of the following: (a) patient baseline biomechanical data for the treatment session, (b) data from previous sessions of the patient, and (c) population data. Then, a baseline beat velocity is selected from the baseline conditions. For example, before playing music, a baseline beat velocity can be selected to match the patient's current pace. Alternatively, the baseline beat velocity can be selected as a fraction or multiple of the patient's current pace. As another option, a baseline beat velocity can be selected to match the baseline beat velocity used in previous sessions with the same patient. As yet another option, the baseline beat velocity can be selected based on baseline beat velocities used for other patients with similar physical conditions. Finally, a baseline beat velocity can be selected based on a combination of any of the above data. A target beat velocity can also be determined based on this data. For example, by referencing improvements observed in other patients in similar situations, the target beat velocity can be selected as a percentage increase in the baseline beat velocity. Velocity is understood to refer to the frequency of the beats in the music.

[0164] In step 1806, the music delivered to the patient from the handheld device 220 on the music delivery device 230 (e.g., earbuds, headphones, or speakers) begins at a baseline tempo or a subdivision of the baseline tempo. To deliver music to the patient at a baseline tempo, music with a constant baseline tempo is selected from a database, or existing music is modified, for example, by selectively speeding up or slowing down, to deliver a beat signal at a constant tempo.

[0165] In step 1808, the patient is instructed to listen to the beat of the music. In step 1810, the patient is instructed to walk at the baseline beat speed, optionally receiving cues about the left and right feet. The patient is instructed to walk so that each step is closely matched to the beat of the music, for example, walking "in time" to the beat. Steps 1806, 1808, and 1810 can be initiated by the therapist or by auditory or visual commands on the handheld device 220.

[0166] In step 1812, sensors 200, 206, and 208 on the patient are used to record patient data, such as heel impact pressure, 6D motion, EMG activity, and video recordings of patient movement. All sensor data is timestamped. Data analysis is performed on the timestamped sensor data, including the “gated” analysis discussed herein. For example, sensor data (e.g., heel impact pressure) is analyzed to determine the start time of each step. Additional data received includes the time associated with each beat signal of the music provided to the patient.

[0167] In step 1814, a connection is made to the attached model (e.g., the integrated machine learning system 410 of analysis system 108 or a model downloaded to collector 106 and running on handheld device 220) for prediction and classification. (It is understood that this connection may be pre-existing or initiated at this time.) This connection is typically very fast or transient.

[0168] In step 1816, the optional entrainment analysis performed by the analysis system 108 is applied to the sensor data. Entrainment analysis includes determining the delay between the beat signal and the start of each step taken by the patient. As output of the entrainment analysis, the accuracy of the entrainment is determined, for example, a measurement of the instantaneous relationship between the baseline speed and the patient's steps, as discussed above regarding entrainment potential and EP ratio. If the entrainment is inaccurate, for example, if the entrainment potential is not constant within tolerance, adjustments are made in step 1818, such as speeding up or slowing down the beat speed, increasing the volume, increasing sensory input, superimposing a metronome or other related sounds, etc. If the entrainment is accurate, for example, if the entrainment potential is constant within tolerance, the speed is incrementally changed in step 1820. For example, the baseline speed of music played on a handheld device is increased towards a target speed, for example, by 5%.

[0169] In step 1822, the connection is made to the entrainment model for prediction and classification. (It should be understood that this connection may be pre-existing or initiated at this time.) In step 1824, an optional symmetry analysis is applied to the sensor data. As the output of the symmetry analysis, the symmetry of the patient's gait is determined, for example, the degree of matching between the patient's left and right foot movements in terms of stride length, speed, stance period, swing period, etc. If the gait is not symmetrical, for example, below a threshold, the music broadcast to the patient is adjusted via a handheld device in step 1826. A first modification can be made to the music played during movement of one of the patient's feet, and a second modification can be made to the music played during movement of the patient's other foot. For example, a minor chord (or increased volume, sensory input, speed variation, or sound / metronome overlay) can be played on one side (e.g., the affected side), while a major chord is played on the other side (e.g., the unaffected side). The machine learning system 410 predicts in advance when symmetry problems will occur based on the "fingerprint" of the scene leading to the symmetry problem, for example, by analyzing movements indicating asymmetry. Asymmetry can be determined by comparing a person’s normal gait parameters with their background, which can determine how one side is affected and compare it with the other side.

[0170] In step 1828, the connection to the clamp model is made for prediction and classification. (It should be understood that this connection may be pre-existing or initiated at this time.) In step 1830, an optional center of balance analysis is performed on the sensor data, for example, whether the patient is leaning forward. This analysis can be performed by combining the outputs of the foot sensors with the video output. As output from the center of balance analysis, it is determined whether the patient is leaning forward. If the patient is leaning forward, in step 1832 the patient is prompted to “stand up,” either by the therapist or via auditory or visual instructions on a handheld device.

[0171] In step 1834, the connection is made to the entrainment model for prediction and classification. (It should be understood that this connection may be pre-existing or initiated at this time.) In step 1836, priming analysis is applied to sensor data, such as the patient exhibiting hesitation or difficulty in initiating walking. As the output of the priming analysis, it is determined whether the patient exhibits priming problems. If the patient exhibits priming problems, for example, below a threshold, tactile feedback can be provided to the patient, which may include a countdown of the beat speed or a countdown before the song begins, as shown in step 1838.

[0172] In step 1840, it may be optionally determined whether the patient is using an assistive device, such as a cane, crutch, walker, etc. In some embodiments, the handheld device 220 provides a user interface for the patient or therapist to input information about the use of the assistive device. If a cane is present, the analysis changes to a three-dimensional approach, such as cane, right foot, left foot, and is prompted by the therapist in step 1842 with “left foot,” “right foot,” and “cane” cues, either provided by the therapist or via auditory or visual instructions on the handheld device.

[0173] In step 1844, the connection to the entrainment model is made for prediction and classification. (It should be understood that this connection may be pre-existing or initiated at this time.) Optional entrainment analysis 1846 is applied to the sensor data, essentially as described above in step 1816, but with the differences noted here. For example, the entrainment may be compared with previous entrainment data from earlier in the session, with data related to the patient's previous sessions, or with entrainment data from other patients. As output of the entrainment analysis, the accuracy of the entrainment is determined, such as how well the patient's gait matches the baseline speed. If the entrainment is inaccurate, adjustments are made in step 1848, essentially in the same manner as in step 1818 above.

[0174] If the bandage is accurate, step 1850 determines whether the patient is walking at the target speed. If the target speed is not reached, the method proceeds to step 1820 (as described above), where the speed is incrementally changed. For example, the baseline speed of the music played on the handheld device is increased or decreased towards the target speed, for example, by 5%. If the target speed has been reached, the patient can continue treatment for the remainder of the session (step 1852). In step 1854, music at the desired speed to be used when not in a treatment session can be... Figure 2 The music is organized and preserved on device 220. This music content is used as homework / exercise for patients between dedicated therapy sessions. At step 827, the procedure ends.

[0175] It is understandable, as stated above, and Figure 18 The steps shown can be performed in a different order than those disclosed. For example, the assessments at steps 1816, 1824, 1830, 1836, 1840, 1846, and 1850 can be performed simultaneously. Furthermore, multiple connections to the analysis system 108 (e.g., steps 1814, 1822, 1828, 1834, and 1844) can be executed at once during the entire treatment session described.

[0176] Figure 19A technique for neglect training is illustrated. For neglect training, the system and method described herein use connected hardware to provide tactile / vibrational feedback when a patient correctly hits a target. The connected hardware includes devices, video motion capture systems, or connected bells. All of these devices are connected to the system described, vibrate upon being struck, and have a speaker to play auditory feedback. For example, a connected bell provides data to the system in the same manner as sensor 200, such as data about the patient's bell-hitting. A video motion capture system provides video data to the system in the same manner as video camera 206. The key input for neglect training is information related to movement tracked to a specific location. The procedure uses connected hardware to provide tactile / vibrational feedback when the patient correctly hits a target. Suitable populations for neglect training include patients with spatial neglect or unilateral visual neglect.

[0177] Figure 19 The flowchart shown is for ignoring training and Figure 18 The flowcharts for gait training shown are essentially the same, but differences are pointed out here. For example, baseline testing determines the patient's condition and / or improvement relative to previous tests. In some embodiments, baseline testing includes displaying four objects evenly spaced from left to right on a screen (e.g., display 222 of handheld device 220). The patient is instructed to strike the objects in time with the beat of background music, either by prompts displayed on display 222 or verbal prompts from the therapist. Similar to gait training, the patient is instructed to strike the objects in time with the beat of background music. Each accurate strike provides feedback. After baseline information is collected, multiple objects evenly distributed from left to right are displayed on the screen. As described above, the patient is instructed to strike the objects sequentially from left to right with the beat of background music. Each accurate strike provides feedback. Similar to gait training, analysis system 108 evaluates and categorizes the patient's responses and provides instructions to increase or decrease the number of objects, or to increase or decrease the tempo of the music to achieve a target speed.

[0178] Figure 20 Techniques for intonation training are illustrated. For intonation training, the systems and methods described in this paper rely on speech processing algorithms. Commonly chosen phrases are frequently used words from the following categories: bilabial consonants, guttural consonants, and vowels. Hardware is connected to the patient to provide tactile feedback to one of their hands at a rate of one minute. Key inputs for intonation training are tone of voice and the spoken word, as well as the rhythm of speech. Suitable candidates for intonation training include individuals with Broca's aphasia, expressive aphasia, non-fluent aphasia, aphasia, autism spectrum disorder, and Down syndrome.

[0179] Figure 20 The flowchart shown is for intonation training and Figure 18The flowcharts for gait training shown are essentially the same, but some differences are noted here. For example, tactile feedback is provided to one of the patient's hands to encourage tapping. The patient is then instructed to listen to playing music via cues displayed on monitor 222 or verbal cues from the therapist. The spoken phrase to be learned is played in two parts, the first part being a high note and the second part a low note. The patient is then instructed to sing the phrase using the two notes being played via cues displayed on monitor 222 or verbal cues from the therapist. Similar to gait training, the analysis system 108 assesses the patient's response and categorizes the response based on the accuracy of the pitch, the words spoken, and the ranking of the response, providing instructions to offer alternative phrases and comparing the response to target speech parameters.

[0180] Figure 21 Techniques for music-stimulated training are illustrated. For music-stimulated training, the systems and methods described herein rely on speech processing algorithms. Familiar songs are used in conjunction with the algorithms to isolate anticipated parts (referred to as anticipatory violations). The hardware includes speakers for receiving and processing the patient's singing, and in some embodiments, the therapist may manually provide input regarding singing accuracy. Key inputs are information related to speech intonation, spoken words, speech rhythm, and musical preferences. Suitable populations include patients with Broca's aphasia, non-fluent aphasia, TBI, stroke, and primary progressive aphasia.

[0181] Figure 21 The flowchart shown is for music stimulation training. Figure 18 The flowcharts shown for gait training are essentially the same, but differences are noted here. For example, a song is played for the patient, and the patient is instructed to listen to the song via cues appearing on display 222 or verbal cues from the therapist. Musical cues are added to the song. Subsequently, at the expected point, if a word or sound is missed, a gestural musical cue is played to prompt the patient to sing the missing word or sound. As with gait training, the analysis system 108 assesses the patient's response and categorizes the response based on the patient's or assistant / therapist's pitch accuracy, the words spoken, and rank, and provides instructions to play additional parts of the song to elevate the speech to the target speech parameters.

[0182] Figure 22Techniques for gross motor training are illustrated. For gross motor training, the systems and methods described herein are designed to assist in motor incoordination, range of motion, or initiation. A more challenging aspect of the exercises is the musical “emphasis,” for example, through the use of melody, harmony, rhythm, and / or dynamic cues. Key inputs are information related to the motion captured in the X, Y, and Z axes via connected hardware or a video camera system. Suitable populations include individuals with neurological, orthopedic, strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

[0183] Figure 22 The flowchart shown is for gross motor training and Figure 18 The flowcharts for gait training shown are essentially the same, but some differences are noted here. Similar to gait training, patients are provided with prompts to move in time with the baseline beat selected by the music. The analysis system 108 assesses the patient's response and categorizes it based on the accuracy of the movements and embellishments as described above, and provides instructions to increase or decrease the speed of the music being played.

[0184] Figure 23 Techniques for grip strength training are illustrated. For grip strength training, the systems and methods described herein rely on sensors associated with a gripper device. The hardware includes a gripper device with pressure sensors and a connected speaker associated with a handheld device 220. The key input is pressure applied by the patient to the gripper device in a manner similar to the heel impact pressure measured by sensor 200. Suitable populations include patients with neurological, orthopedic, strength, endurance, balance, posture, range of motion, TBI, SCI, stroke, and cerebral palsy.

[0185] Figure 23 The flowchart shown is for grip strength training. Figure 18 The flowcharts for gait training shown are essentially the same, but some differences are noted here. Similar to gait training, the patient is prompted to apply force to the gripper device in a timely manner in sync with the baseline beat of the selected music. The analysis system 108 assesses the patient's response, categorizes the response based on the actions and gripping accuracy as described above, and provides instructions to increase or decrease the speed of the music playback.

[0186] Figure 24 Techniques for voice prompt training are illustrated. For voice prompt training, the systems and methods described herein rely on speech processing algorithms. Hardware may include a speaker for receiving and processing a patient's singing, and in some embodiments, the therapist may manually provide input regarding speech accuracy. Key inputs are intonation and spoken words, as well as speaking rhythm and musical preferences. Suitable populations include patients with robotic, word-checking, and stuttering problems.

[0187] Figure 24The flowchart shown is for voice prompt training and Figure 18 The flowcharts for gait training shown are essentially the same, but differences are noted here. Similar to gait training, the patient is prompted to speak sentences by saying one syllable in time with each beat of the music, either through cues displayed on monitor 222 or verbal cues from the therapist. Analysis system 108 evaluates the patient's speech and, as described above, categorizes responses based on the accuracy of speech and enunciation, providing instructions to increase or decrease the speed of the music playback.

[0188] Figure 25 A technique for training minimally conscious patients is illustrated. The system and method described herein rely on an imaging system, such as a 3D camera, to measure whether the patient's eyes are open, the direction the patient is looking, and consequently, the patient's pulse or heart rate. The procedure searches for and optimizes heart rate, stimulation, respiratory rate, eye closure, posture, and restlessness. Suitable populations include comatose and impaired consciousness patients.

[0189] Figure 25 The flowchart shown is for training minimally conscious patients and Figure 18 The flowcharts for gait training shown are essentially the same, but differences are noted here. Similar to gait training, increasing stimuli are provided to the patient at their breathing rate (PBR). For example, stimuli are first provided to the patient at a musical chord PBR, and the patient's eyes are observed to open. If the patient's eyes are not open, the stimuli are progressively increased, from humming a simple melody at the PBR, to singing "aah" at the PBR, to singing the patient's name at the PBR (or playing a recording of such a sound), with the patient's eyes checked at each input. The analysis system 108 assesses the patient's eye tracking and categorizes responses according to level of consciousness, providing instructions to change the stimuli.

[0190] Figure 26-28 Techniques for attention training are illustrated. For attention training, the systems and methods described herein operate in a closed-loop manner to help patients maintain, segment, alternate, and select attention. No visual cues are allowed to signal which movements to perform. Suitable candidates include patients with brain tumors, multiple sclerosis, Parkinson's disease, neurological disorders, and injuries.

[0191] Figure 26 The flowchart shown is for continuous attention training and Figure 18The flowcharts for gait training shown are essentially the same, but the differences are pointed out here. Similar to gait training, the patient is provided with an instrument (e.g., any instrument will work, such as drumsticks, drums, keyboards, or their respective wireless versions), and instructed to follow or perform actions such as those indicated by prompts appearing on display 222 or verbal prompts from the therapist. Figure 26 The tasks are defined by audio prompts at levels 1 through 9. Analysis system 108 assesses the patient's ability to accurately complete the tasks and categorizes responses to adjust the task's speed or difficulty. Similarly, Figure 27 A flowchart for alternating attention training is shown, in which instructions are provided to follow or perform tasks that alternate between the left and right ears via cues appearing on display 222 or verbal cues from the therapist. Figure 28 A flowchart for distraction is shown, in which instructions are provided to follow or perform tasks based on audio cues from audio signals in the left and right ears.

[0192] Figure 29 The flowchart shown is for flexibility training and Figure 18 The flowcharts for gait training shown are essentially the same, but differences are pointed out here. For flexibility training, patients are instructed to tap the piano keys with their fingers to collect baseline movement and range of motion information. The song begins with a specific beat per minute, and the patient begins tapping with their baseline hand index. The analysis system 108 assesses the patient's ability to accurately complete the task and categorizes the response to adjust the speed or difficulty of the task.

[0193] Figure 30 The flowchart shown is for oral motor training. Figure 18 The flowcharts shown for gait training are essentially the same, but differences are pointed out here. For oral motor training, patients are instructed to perform tasks alternating between two sounds (e.g., “ooh” and “aah”). The analysis system 108 assesses the patient’s ability to accurately complete the tasks and categorizes the responses to change the speed or difficulty of the tasks, for example, by providing different target sounds.

[0194] Figure 31 The flowchart shown is for breathing training and Figure 18 The flowcharts for gait training shown are essentially the same, but the differences are pointed out here. For breathing training, a baseline respiratory rate and shallowness of breathing are determined. The music is set to a baseline tempo based on the patient's breathing rate, and the patient is instructed to perform... Figure 31 The system describes the breathing task at the level described in the analysis system 108. The analysis system 108 assesses the patient's ability to accurately complete the task and classifies the response to change the speed or difficulty of the task, for example, by providing different breathing patterns.

[0195] This document further describes methods, systems, and apparatuses for using augmented reality (AR) and augmented audio (AA) to support next-generation medical and therapeutic systems to improve or maintain motor function. Exemplary embodiments of the augmented neurologic rehabilitation, recovery, or maintenance (“ANR”) systems and methods disclosed herein are based on entrainment techniques, utilizing additional sensor streams to determine therapeutic benefits and inform closed-loop treatment algorithms. Exemplary systems and methods for neurorehabilitation, which can be used to implement embodiments of ANR systems and methods, are illustrated and described above and in U.S. Patent Application No. 16 / 569,388, “Systems and Methods for Neurologic Rehabilitation,” co-pending and co-assigned by McCarthy et al., which is a continuation of U.S. Patent No. 10,448,888, entitled “Systems and Methods for Neurologic Rehabilitation,” issued October 22, 2019. U.S. Patent No. 10,448,888 is based on and claims priority to U.S. Provisional Patent Application No. 62 / 322,504, entitled “Systems and Methods for Neurologic Rehabilitation,” filed April 14, 2016. All documents are incorporated herein by reference, as if they were set forth in their entirety herein.

[0196] According to one or more embodiments, the ANR system and method may include a method for providing an AR3D dynamic model of a specific person or object, the method comprising retrieving images or videos of the person or object by querying a cloud database and a local database. The ANR system and method may also be configured to leverage the neuroscience of the ability of music to improve motor function and the neuroscience of how visual images affect recovery (e.g., the use of mirror neurons), fusing and / or synchronizing the dynamic model, the patient or human, audio content, and contextual information about the natural environment into a synchronized state.

[0197] According to one or more embodiments, the ANR system and method include a method for incorporating AA techniques for repetitive motor activities. Augmented audio (AA) combines real-world sounds with an additional computer-generated audio “layer” that enhances sensory input. At the heart of rhythmic neuroscience is the use of stimuli to engage the motor system in repetitive motor activities, such as walking. Adding AA to a treatment regimen can enhance therapeutic effects, increase adherence to the regimen, and provide greater safety in the form of enhanced patient contextual awareness. The disclosed embodiments for adding AA can utilize the neuroscience of music to blend numerous audio signals, including external natural environmental sound input, recorded content, rhythmic content, and voice guidance, into a synchronized state. Furthermore, it can include the ability to combine algorithmically generated music with underlying rhythmic cues, as described in

[00218] for patients with motor or physical disabilities. This can be achieved by fusing such generated rhythms with input from real-time biometric data from the patient into an interactive feedback state. An exemplary system and method for generating auditory stimulation for neurorehabilitation by means of an algorithm is illustrated and described in co-pending and co-assigned U.S. Patent Application No. 16 / 743,946, filed January 15, 2020, entitled “ENHANCING MUSICFOR REPETITIVE MOTION ACTIVITIES”, filed by McCarthy et al., which is a continuation of U.S. Patent Application No. 16 / 044,240, filed July 24, 2018, and now U.S. Patent Application No. 10,556,087, also entitled “ENHANCING MUSICFOR REPETITIVE MOTION ACTIVITIES”, published February 11, 2020. U.S. Patent Application No. 10,556,087 claims priority under Section 119(e) of the U.S. Patent Act in U.S. Provisional Application No. 62 / 536,264, filed July 25, 2017. The entire contents of all these documents are incorporated herein by reference as if they were set forth in their entirety herein.

[0198] Neuroplasticity, entrainment, and mirror neuron science are fundamental scientific components supporting the disclosed embodiments. Entrainment is the term for the activation of the brain's motor centers in response to external rhythmic stimuli. Research indicates that audiomotor pathways exist in the spinal reticular formation, the part of the brain responsible for movement. Initiation and timing of movement through these pathways demonstrate that the motor system can couple with the auditory system to drive movement patterns (Rossignol and Melville, 1976). Entrainment processes have been shown to effectively improve walking speed (Cha, 2014), reduce gait variability (Wright, 2016), and reduce the risk of falls (Trombetti, 2011). Neuroplasticity refers to the brain's ability to strengthen pre-existing neural connections, allowing individuals to acquire new skills over time. Research shows that music promotes changes in certain motor areas of the brain, suggesting that music can promote neuroplasticity (Moore et al., 2017). Mirror neurons are activated when you perform one action and when you observe another. Therefore, when you see someone else perform an action, your brain reacts as if you were the one performing that action. This allows us to learn behavior through imitation. This is important in the context of the disclosed embodiments because when patients observe augmented reality human simulations, they can use mirror neurons to mimic their actions.

[0199] According to one or more embodiments, the ANR system and method are configured to process images / videos, removing / adding people / objects smaller than or larger than a specified size from the images / videos in response to patient or therapist incidents received as input to the ANR system. Such incidents can be patient responses (e.g., instructions to reduce scene complexity) or therapist instructions to introduce occlusions of people / objects that may increase scene complexity. Embodiments may support recording all data related to all patient or therapist incidents in addition to the session data itself.

[0200] According to one or more embodiments, the ANR system and method include a telepresence approach that allows therapists to link to a remote patient using the system. In addition to fully supporting all local features experienced by the patient and therapist in the same location, the telepresence approach includes biomechanical motion tracking of the patient relative to an AR3D dynamic model of the person / object.

[0201] The AR3D dynamic model is a software-based algorithmic process within an ANR system and method for generating AR / VR visual scenes presented to patients. These scenes are animated based on fundamental principles of neuroplasticity, entrainment, and mirror neuron science to facilitate outcomes for clinical or training goals. The telepresence method is configured to provide the ability to operate an interactive video link between a therapist and a remote patient using this invention. The telepresence method of this invention supports projecting images / videos from a remotely located patient to indicate the patient's relative position within the AR3D dynamic model. The telepresence method also provides therapists with the ability to adjust the AR model in real time (spatial location or which items are available) and allows for modification of the session. It will also allow therapists to view the relationship between the patient and the model.

[0202] Figure 32 A conceptual overview of the main components of an exemplary ANR system 3200 is depicted, which uses closed-loop feedback that measures, analyzes, and acts on a person to facilitate outcomes toward clinical or training goals. It should be understood that the ANR system 3200 can be implemented using various hardware and / or software components of the system 100 described above. As shown, the ANR system measures or receives inputs related to gait parameters, the natural environment, environment / user intent (including past performance), physiological parameters, and real-time feedback (e.g., closed-loop, real-time decision-making) of outcomes. One or a combination of these inputs can be directed to a Clinical Thinking Algorithm (CTA) module (CTA) 3208, which controls the analysis and application of that information. In some embodiments, an example of real-time feedback would be that CTA 3208 determines that the quality of gait attitude quantities measured by the user (e.g., symmetry, stability, and gait cycle time variations) has exceeded a safety threshold of the module, thereby triggering a new cue response. Such cue responses are visual and audio, for example, by adding a metronome audio layer and modifying the AR scene to increase the salience of the beat in music to simulate or guide the user into a safer gait speed and movement behavior.

[0203] In CTA module 3208, the actions performed by the ANR system are defined based on input analysis. Actions can be output to the patient in the form of various types of stimuli, including music (or other components), rhythm, ultrasound processing of sound, spoken words, augmented reality, virtual reality, augmented audio, or haptic feedback. For example... Figure 32As shown, judgments made by a critical thinking algorithm related to various inputs, results, etc., are provided to the AR / AA output modeling module 3210, which is programmed to dynamically generate / modify the patient's output. The output is provided to the patient via one or more output devices 3220 (such as visual and / or audio output devices and / or haptic feedback devices). For example, AR visual content can be output to AR glasses 3222 worn by the patient. Enhanced audio content can be provided to the patient via audio speakers or headphones 3225. As will be understood, other suitable visual or audio display devices can be used without departing from the scope of the disclosed embodiments. Furthermore, although in Figure 32 The individual components of the system are shown separately, but it should be understood that the features and functions of various aspects of the system can be combined.

[0204] The Clinical Thinking Algorithm Module 3208 in this medical / therapeutic system can be configured to implement critical thinking algorithms focused on the recovery, maintenance, or enhancement of motor function, including but not limited to upper limb, lower limb, agitation, postural stability, foot drop, dynamic stability, panting, oral motor function, respiration, endurance, heart rate training, respiratory rate, optical flow, boundary support training, strategy training (ankle, knee, and hip), attractor coupling, muscle firing, training optimization, and gait. One or more of these CTAs can also be implemented concurrently or in combination with other interventions with similar objectives. Examples may include implementing CTAs in conjunction with functional electrical stimulation, deep brain stimulation, transcutaneous electrical nerve stimulation (TENS), gamma frequency audio clipping (20-50 Hz), or other electrical stimulation systems. Furthermore, the administration and manipulation of CTAs can be combined with the administration of anticonvulsant drugs or neurotoxin injections. For example, CTAs can be applied or initiated during or before the time window in which these interventions have shown peak effectiveness, as a method of activating the motor system.

[0205] ANR system input

[0206] The inputs to the ANR System 3200 are crucial for enabling the system to measure, analyze, and operate in a continuous, cyclical manner, which helps achieve clinical or training goals.

[0207] Input received at the system may include measuring human motion via sensors to determine biomechanical parameters of the motion (e.g., temporal, spatial, and left / right comparisons). These motion sensors can be placed anywhere on the body and can be a single sensor or an array of sensors. Other types of sensors may be used to measure other input parameters, which may include respiratory rate, heart rate, oxygen level, temperature, electroencephalogram (EEG) for recording spontaneous electrical activity in the brain, electrocardiogram (ECG or EKG) for measuring cardiac electrical activity, electromyogram (EMG) for assessing and recording electrical activity generated by skeletal muscle, photoplethysmogram (PPG) for detecting changes in blood volume in the microvascular bed of tissue, typically using pulsatile oximeters for measuring changes in skin light absorption, optics, inertial measurement units, video cameras, microphones, accelerometers, gyroscopes, infrared, ultrasound, radar, RF motion detection, GPS, barometers, RFID, radar, humidity sensors, or other sensors that detect physiological or biomechanical parameters. For example, in Figure 32 In the exemplary ANR system 3200 shown, one or more sensors 3252 (e.g., IMU, footpad sensor, smartphone sensor (e.g., accelerometer), and environmental sensors) can be used to measure gait parameters. Additionally, one or more sensors 3254 (e.g., PPG, EMG / EKG, and respiratory rate sensors) can be used to measure physiological parameters.

[0208] Furthermore, environmental information regarding expected results, usage environment, past meeting data, other technologies, and other environmental conditions can be received as input to the CTA module and used to adjust the CTA's response. For example, such as Figure 32 As shown, environmental information 3258 and environmental input 3256 can be received as inputs to further notify the CTA 3208 of its operations. An example of using environmental information is that information from past user gait patterns can be combined with artificial intelligence (AI) or machine learning (ML) systems to provide patients with more personalized clinical goals and actions. These goals can modify target parameters such as steps per minute limit, walking speed, heart rate variability, oxygen consumption (VO2max), respiratory rate, session length, asymmetry, variability, walking distance, or desired heart rate.

[0209] Furthermore, Bluetooth Low Energy (BLE) beacons or other wireless proximity technologies (such as wireless triangulation or angle of arrival) can be used to detect environmental information, enabling wireless positioning triggers to make a person / object appear and / or disappear in the patient's field of vision relative to the AR3D dynamic model based on the detected location. In some embodiments, the AR3D dynamic model output by the ANR system can be controlled by the therapist and / or beacon triggers to alter or maintain the patient's navigation requirements. These triggers can be used in conjunction with gait or physiological data as described above to provide additional triggering beyond wireless beacon triggering. For example, gait data feedback from an IMU product allows for a gait feedback loop that provides the ANR system 3200 with the ability to implement changes during the AR3D dynamic model software process.

[0210] Clinical reasoning algorithm

[0211] According to one or more embodiments, the CTA module 3208 implements a clinical reasoning algorithm, which is configured to control the applied treatment to facilitate outcomes toward clinical or training goals. Clinical goals may include... Figures 18 to 31 Related projects, and as further examples, include anxiety interventions and training / physical activity goals for Alzheimer's disease, dementia, bipolar disorder, and schizophrenia. This section discusses various non-limiting exemplary techniques that can be used to provide appropriate rehabilitation responses determined by CTA, such as modulating pacing and synchronizing AR visual scenes. Each of these techniques can be implemented using independent CTAs or in combination with each other. In one or more embodiments, system 3200 can be configured to combine CTAs with entrainment principles for repetitive motor activities, and in other cases, they can be combined with each other toward other goals.

[0212] As an example, and not a limitation, the CTA module 3208 of the ANR system 3200 can be configured to create a virtual treadmill output via an AR / AA output interface, utilizing a combination of biomechanical, physiological data, and environmental factors. While the treadmill maintains a person's pace through the movement of the physical belt, the virtual treadmill dynamically adjusts using CTA based on the belt-grip principle, independently regulating the person's walking or movement speed, similar to other exercise interventions. However, in addition to or instead of using rhythmic stimuli to drive an individual toward a biomechanical goal as described above, the virtual treadmill can be generated and dynamically controlled based on the patient's toward target parameters (such as those listed above as target parameters).

[0213] Target parameters can be set based on inputs from clinicians, users, historical data, baseline conditions, or beyond clinically significant points. Additionally, target parameters can be set based on or follow recommendations for specific exercise length, duration, and intensity for certain conditions (such as heart disease, asthma, chronic obstructive pulmonary disease (COPD), fall prevention, musculoskeletal disorders, osteoarthritis, and general aging).

[0214] This document further describes an exemplary embodiment of the ANR system 3200, wherein the CTA module 3208 is configured to utilize biomechanical data, physiological data, and the environment to provide gait training therapy in the form of a virtual treadmill and rhythmic auditory stimulation output via the AR and AA output interfaces 3220.

[0215] Figure 37 This is a process flowchart illustrating an exemplary routine 3750 for providing gait training therapy to a patient using the ANR system 3200. Figure 38 This is a hybrid system and process diagram conceptually illustrating aspects of an ANR system 3200 for implementing a gait training routine 3750 according to exemplary embodiments of the disclosed subject matter. As shown, sensors 3252, particularly foot-mounted IMUs, capture sensor data related to gait parameters provided to a CTA module 3208. Furthermore, an AA / AR modeling component 3210, including an audio engine, receives input from the CTA and is configured to generate a set of audio cues including rhythmic music and prompting content, interactive voice guidance, and spatial and audio effects processing. Similarly, an AA / AR modeling component 3210, including an AR / VR modeling engine (also known as an AR3-D dynamic model), is shown receiving input from the CTA and is configured to generate a set of visual cues including virtual AR actors and objects (e.g., virtual human walking), background motion animation (e.g., virtual treadmill, footsteps / footprints, and animations), and scene lighting and shadows.

[0216] Figure 39 This is a conceptual, more detailed illustration of a hybrid system and process diagram of an exemplary audio output device 3225 and an enhanced audio generation component of the ANR system 3200. (See diagram for more details.) Figure 39As shown, in one embodiment, the AA device may use, for example, a stereo microphone to capture ambient sound. The AA device may also use a stereo transducer to generate audio output. The AA device may also include a head-mounted IMU. As will be understood, the AA device may also include audio signal processing hardware and software components for receiving, processing, and outputting enhanced audio content received alone or in combination with other content such as ambient sound from the AA / AR module 3210. As shown, the CTA module 3208 receives gait parameters, including gait parameters received from sensors including a foot-mounted IMU and a head-mounted IMU. Furthermore, in one embodiment, the CTA receives data related to physiological parameters from other sensor devices such as a PPG sensor.

[0217] Now back Figure 37 In step 3700, while the ANR system 3200 calibrates and collects preliminary gait data (such as stride length, speed, gait cycle time, and symmetry), the patient wearing the AR / AA output device 3220 and IMU sensor 3252 begins walking. In step 3701, the CTA module 3208 determines the baseline tempo rate of both the music playback and the virtual AR scene to be displayed. For example, as combined Figure 18 The baseline tempo can be determined by CTA.

[0218] Using the user's gait cycle time as input, in step 3702, the audio engine (i.e., the audio modeling component of the AR / AA modeling module 3210) generates corresponding tempo-adjusted music and any supplementary rhythmic cues, such as metronome sounds, to enhance beat salience.

[0219] In step 3703, the visual AR engine (i.e., the visual modeling component of the AR / AA modeling module 3210) generates a mobile virtual scene, such as a scene as understood in the video game industry. More specifically, in one embodiment, the virtual scene includes visual elements presented under the control of the CTA and shares a common timing reference with the audio engine to synchronize the elements of the visual scene with the music and tempo. Although the AR scenes described herein include virtual treadmills or virtual people or footsteps, AR scenes can be any or more of the various examples discussed herein, such as virtual treadmills, walking virtual people, virtual crowds, or dynamic virtual scenes.

[0220] In steps 3704 and 3705, music / rhythm and visual content are delivered to the patient using an AR / AA device 3220, such as a lightweight heads-up display (e.g., AR goggles 3222) with headphones 3225. Under the control of the CTA, the patient receives instructions regarding the treatment in steps 3706 and 3707 via voice-over cues generated by an audio engine. This may include a pre-training preview to allow the patient to become accustomed to and practice the visual landscape and audio experience.

[0221] Figure 40 An exemplary AR virtual scene 4010 presented for a patient to wear is shown. As illustrated, the scene may include an animated 3D image of another person walking "in front" of the patient, their steps and walking movements synchronized with the tempo of music. More specifically, in one embodiment, the AR actor walks at the same speed as the baseline beat speed of the audio content generated by the CTA and audio engine. In this example, the patient's goal may be to match their steps rhythmically to the audio and visually to the actor. Furthermore, as... Figure 40 As shown, an AR scene can include multiple footsteps, with additional cues such as L and R indicating the left and right feet.

[0222] The scene, including footsteps, can virtually move towards the patient at a prescribed rate, while a virtual actor walks in front of the patient in a direction away from the patient. In one embodiment, the scene can move according to the gait cycle time, where GCT (right foot) / 2*60 = music tempo defined by CTA. Furthermore, additional cues generated in conjunction with the AR scene can include rhythmic audio cues that reinforce visual cues. For example, an effective reinforcement method could include the AA system 3210 generating the footsteps of a virtual actor in rhythmic synchronization, simulating a team marching to a common beat. By providing the patient with rhythmic audio components generated based on visual stimuli, and timing the movement of visual elements with rhythmic audio stimuli, the system further reinforces the virtuous cycle between the fundamental therapeutic concepts of entrainment and mirror neurons.

[0223] Through further examples, Figure 41 An exemplary AR virtual scene 4110 presented to a patient is shown. Figure 41 As shown, a virtual treadmill can be generated and dynamically controlled based on the patient's cues toward target parameters (such as those listed above). In this example, the generated AR treadmill animates the movement of the treadmill surface 4115 and generates virtual steps at the same speed defined by the CTA for the auditory stimulus. Furthermore, the 3D animation of the virtual treadmill can include visually highlighted steps or tiles that the patient can use as visual targets, while cues generated by the CTA control. Therefore, in this example, the patient's goal is to rhythmically match their steps to the audio and visually match the animated target steps. Figure 41As further illustrated herein, animated steps 4120 can be displayed on the surface of the virtual treadmill 4115, moving in the direction towards the patient as indicated by directional arrows, which can be reversed if the patient is performing backward walking training exercises. Furthermore, animations highlighting target steps 4125L (left foot) and / or 4125R (right foot) are highlighted in sync with the rhythm of the CTA to prompt the user to step with the corresponding foot (e.g., left or right foot). Additionally, as further described herein, dynamic adjustments such as... can be made based on corresponding changes in the patient's entrained potential (EP) and the audio stimulus. Figures 40-41 The virtual scene shown is an example. Other adjustments to the virtual scene can include changing the virtual background environment to simulate different walking scenarios, weather, surfaces, lighting, and tilt.

[0224] Now back Figure 37 When the patient begins to walk, in step 3708, a biomechanical sensor (e.g., sensor 3252) measures real-time data to assess the patient's entrainment level. In step 3709, the CTA module 3208 determines the entrainment potential (e.g., as shown in the image). Figure 18 (As shown), and used to determine how to achieve the training session objectives. As discussed in previous embodiments of this disclosure, the entrainment potential can be the basis for modifying rhythmic audio stimuli and visual scenes (occurring in step 3710). For example, CTA analyzes the history of input data on the patient's gait cycle time and compares it with the rhythmic intervals of the beats transmitted to the patient by the audio device. Exemplary methods for modifying audio stimuli based on entrainment potentials have been similarly described above. In one example, if the EP value calculated for the patient's steps over a period of time is not within the specified range of acceptable EP values ​​and / or is not sufficiently consistent, the CTA module can instruct the AA / AR modeling module to adjust (e.g., reduce) the speed of the RAS and accordingly adjust the motion speed of the AR scene in sync with the RAS. In another example, if sufficient step time is in phase with the beat time, then the patient is considered to be CTA entrained.

[0225] Once the entrainment is complete, in step 3711, the CTA assesses whether the patient has reached the target. If the target has not been reached, one or more target parameters can be adjusted in step 3712. For example, in one example, the CTA compares the RAS velocity and the associated AR scene velocity with target velocity parameters (e.g., training / treatment goals) before changing the tempo velocity and / or scene motion velocity in consideration of comparisons. Exemplary methods for modulating rhythmic auditory stimulation based on the entrainment potential, which can be implemented by the CTA module 3208, are as described above. Figure 18 Show and describe.

[0226] For example, if the patient has not yet reached their training speed target, modifying the target parameters could include increasing or decreasing the music tempo in step 3712. This would use the RAS mechanism to drive the patient to walk faster or slower. Alternatively, another training goal could be to increase the patient's stride length, which could be achieved by slowing down the motion speed parameters of the image. By modifying the visual scene, the patient would be driven to use a mirror motion mechanism to simulate the visual example presented to them.

[0227] It is important to understand that audio and visual outputs are mutually reinforcing stimuli: the visual scene and rhythmic stimulus are synchronously layered. Depending on the selected procedure (e.g., CTA), the CTA module 3208 dynamically adjusts the visual scene and rhythmic speed to meet therapeutic goals.

[0228] The above example illustrates how the ANR system 3200 using the CTA module 3208 controls the synchronization of music tempo and AR scene based on biomechanical sensor input to facilitate gait training. It should be understood that the principles of this embodiment are applicable to many disease indications and rehabilitation scenarios.

[0229] In one exemplary configuration, the ANR system 3200 can generate a virtual treadmill to regulate the patient's walking towards a target parameter for oxygen consumption. In this example, the virtual treadmill is generated and controlled to adjust the walking speed to a target parameter for oxygen consumption or efficiency using a treadmill belt. Figure 33 It is a graphical visualization of a real-time session performed using the ANR system 3200, where V02max is used as the target parameter, velocity variation is used as a temporary target, and entrainment is used to drive physiological changes associated with V02max. Figure 33 An example of how this process works in real time is shown. More specifically, Figure 33This is a graphical user interface illustrating various significant data points and parameter values ​​measured, calculated, and / or adjusted in real time by the ANR system during a session. As shown, the top of the interface displays a graph of the entrainment potential values ​​calculated in real time for each step throughout the session. In particular, the top bar graph shows the individual EP calculated for each step, which in this case is the phase correlation between the step time interval and the beat time interval. The central area around EP=1 indicates steps with sufficient entrainment in the beat, or in other words, steps with EP values ​​within the specified range. The next window provides a status bar indicating whether the parameters are within safe ranges. The next window displays the real-time response driven by CTA, particularly based on measured parameters, entrainment, and other aforementioned inputs and feedback from CTA. In particular, circle icons represent algorithmic responses, which include changes in velocity and rhythmic stimulus levels (e.g., volume). The next column only shows velocity and velocity changes themselves. The next window shows the real-time velocity of the rhythmic stimulus provided to the patient over time based on the CTA response. The bottom window displays the measured oxygen consumption and target parameters over time. Although in Figure 33 Although not shown, it should be understood that augmented reality scenes (e.g., a virtual treadmill) can be presented to patients, where visual elements are animated in sync with rhythmic stimuli and dynamically adjusted in sync with real-time adjustments to the speed of the rhythmic stimuli. Examples of how the AA / VR module 3210 can be configured to synchronize visual animation speed and audio can include defining a relationship between the displayed rate of repetitive movement and the speed of the audio cues. For example, based on the beat rate, the treadmill speed and stride length are calculated to define the relationship between the audio and visual elements. Furthermore, the treadmill's reference position, the timing of the steps, and any beat-time animations are synchronized with the output time of the beats, including the beat rate. Using time scaling and video frame interpolation techniques known in the animation industry, a wide range of synchronized virtual scenes can be generated on demand by the AA / VR module 3210 programmatically based on the defined relationship between the audio and visual elements.

[0230] Figure 34 It is a graph depicting the metabolic changes of 7 patients during the first training period (represented by the corresponding set of two points connected by a dashed line). Figure 34 The data supports the idea that intentional entrainment can improve an individual's oxygen consumption. In this case, the graph shows the oxygen consumption (ml oxygen / kg / m) of a person before and after tempo-based training using the ANR system. This figure shows an average reduction of 10%. The results indicate that entrainment can improve endurance and reduce energy expenditure during walking.

[0231] The above process can be similarly implemented for each of the various possible target parameters (e.g., oxygen consumption can be replaced by a substitute target, such as heart rate), and can be used for walking or combined with... Figures 18 to 31 Other interventions discussed.

[0232] According to one or more embodiments, the ANR system 3200 can be configured to compare real-time measurements of a person's motion with the composition of AR images and / or musical content output to the patient during a therapeutic session (e.g., instantaneous velocity, rhythm, harmony, melody, etc.). The system can use this parameter to calculate entrainment parameters, determine phase entrainment, or establish baselines and carrying characteristics. For example, the AR image can move at the same speed or pace as the target parameter. Alternatively, the AR-related motion of the image can be entrained into rhythmic stimuli in sync with how the person should move.

[0233] An example of AR3D dynamic model output could include a projection therapist (virtual actor) or a person walking in the patient's field of vision (virtual actor), initiated by the person performing the treatment (real therapist). For example, Figure 35 This illustration shows a view of a therapist or coach projected onto a patient or trainer using, for example, AR glasses known in the art via AR projection. This AR 3D dynamic model is controlled by one or more CTAs. In the combination of CTAs, the virtual therapist can... Figure 22 The methods shown and described begin, and then they are subjected to gait training programs, such as those combining... Figure 18 As shown and described. Alternatively, both tasks can be performed simultaneously. In these cases, the virtual actor can be controlled by the system to walk or move backward or forward with smooth movements similar to the unaffected side of a patient. This may activate mirror neurons, which are where affected neurons are "encouraged" to mirror the unaffected neurons. The process may also include providing audio stimulation to synchronize the virtual and / or real person with the stimulation.

[0234] In another example, the AR3D dynamic model can be configured to simulate a scene where a patient is walking among or around a crowd and / or with objects in front of and / or to the side of the patient. For example, Figure 36A This diagram illustrates a view of a crowd projected in front of a patient via AR. The system can be configured to project crowds or individuals moving faster or slower than a person's baseline to encourage them to move or stop / start at similar speeds in a real-world, natural environment. The crowd or individuals may be guided by rhythmic auditory stimuli or the beat of other desired targets. The AR3D dynamic model can initiate navigation of varying difficulty. It should be understood that the AR3D dynamic model can be used to dynamically adjust the AR view of the therapist, crowds, individuals, obstacles, etc., based on the CTA output.

[0235] In another example, the AR3D dynamic model can be configured to simulate a patient walking in or around a cone arrangement that provides virtual obstacle heading for patient navigation. The cones are normal obstacles in a therapeutic environment; however, other embodiments can be configured to simulate normal activities of daily life (e.g., grocery shopping). These cones and virtual obstacles can encourage changes in direction: walking sideways and backwards, not just forwards. Here, a wireless beacon trigger can also be used to make the ANR system present appearing and / or disappearing cones. The beacon will be triggered based on detecting the person's position relative to the cone. Furthermore, different levels of difficulty can be set in terms of navigation time and length. The target parameter in this example could be a measure of walking speed or walking quality. Successful navigation would be navigating around the cones without actually touching them. The system can be configured to present a more difficult level (e.g., more obstacles and faster speeds), as long as the person successfully avoids the obstacles and the walking quality is not diminished (measured by increased variability or worsened asymmetry).

[0236] In another example, the AR3D dynamic model can be configured to simulate a patient walking in or around caves and / or cliffs, potentially including obstacles with realistic effects. Realism will enhance the detail required for navigation relative to existing use cases. In another example of a person with an asymmetrical gait pattern, a winding path can be presented when a longer step is required on the affected side. This winding path could also be a separate cliff that must be crossed across a valley without falling. Wireless beacon triggers can be used to make the ANR system appear and / or disappear cave and / or cliff obstacles, thus altering the difficulty of navigation time and path length. The system can use sensor data to synchronize motion to the winding path. The patient's navigation requirements may be a biomechanical response to navigational changes within a baseline-prescribed heading. The system is configured to allow wireless spatial and temporal beacon triggers to influence changes in the AR3D dynamic model. The temporal aspect of these wireless triggers is their ability to turn them on and off. This will allow for maximum flexibility in writing navigation paths for the heading the patient should take as part of a treatment session. The target parameter at that moment is a measure of walking speed or walking quality. Successful navigation will be navigating on the path without stepping off it or falling off a cliff. The system can be configured to present a more challenging level (e.g., more obstacles and faster speeds), as long as the person successfully stays on the path and the quality of walking is not diminished (measured by increased variability or worsened asymmetry).

[0237] In another example, the AR3D dynamic model can be configured to simulate a scenario where a patient stands or sits still, but is instructed to move as virtual objects appear and approach each foot. For example, Figure 36BThis image shows a view of footprints projected in front of a patient via AR. The ANR system can generate a virtual scene where an object can approach the patient's left or right side to encourage lateral walking. The object will be presented as approaching the patient at a predefined speed or beat, following a path such as... Figure 22 The decision tree described in the text. Visual representations of correct actions learned by therapists or patients from past treatments can also be projected.

[0238] In another exemplary AR3D dynamic model implementation, the ANR system can be configured to incorporate haptic feedback into the therapy. For example, the system can use haptic feedback as a signal if the user gets too close to an object or person in the projected AR environment. Rhythmic haptic feedback can also be synchronized with auditory cues to amplify sensory input. AR can also be adaptively and independently enabled to indicate the onset of movement, for example, during gait freeze in a Parkinson's patient.

[0239] In another exemplary AR3D dynamic model implementation, the ANR system can be further configured to combine optical and head tracking. This tracking can be incorporated as feedback to the ANR system, which is configured to trigger auditory input in response to the position of their eyes or head. For example, a person with left-sided neglect who only interacts with the right side of their natural environment could have their left hemisphere of the natural environment occupied by eye and head tracking, providing input and triggering the system to generate auditory cues to shift more attention to the left hemisphere. This data can also be used to track progress over time, as clinical improvement can be measured by the level of awareness in each hemisphere. Another example of this is for people with oculomotor disorders, where left-to-right visual scanning can be improved by performing visual scans in accordance with external auditory rhythms.

[0240] In another exemplary AR3D dynamic model implementation, the ANR system can be configured to provide digital representations of past sessions to show user improvements. These models can be replayed after a session to compare the lifecycle of different sessions or treatments. Digital representations of past (or enhanced) sessions, when paired with audio input for that session, can be used as mental visualization tasks to practice and limit fatigue between walking sessions. The model will display differences in walking speed, cadence, stride length, and symmetry to help show how the user changes over time and how treatment may improve their gait. Therapists can also use this presence before a session begins to help prepare training plans or techniques for subsequent sessions. Researchers and clinicians can also use this modeled presence to better visualize and reactivate 3D images of the evolution of patient progress.

[0241] In another exemplary AR3D dynamic model implementation, an AR / VR environment synchronized with music content can create different walking or dance patterns, including elliptical, spiral, serpentine, intersecting paths with others, and dual-task walking. Dance rhythms such as tango have been shown to have benefits of neurologic music therapy (NMT) and reflexology (RAS) that can be applied to the whole body.

[0242] According to one or more embodiments, an ANR system can be configured to leverage AA techniques to enhance the entrainment process, provide environmental context for the person, and assist the AR experience. To enhance the recovery process, the system can be configured to generate exemplary AA experiences, further described herein, based on inputs obtained from the environment, sensor data, the AR environment, entrainment, and other methods.

[0243] One example of AA (Audio-Awareness) in therapeutic / medical use cases is addressing safety concerns and mitigating risks for patients undergoing therapeutic exercises. An ANR system can be configured to improve contextual awareness while listening to music through headphones by instantaneously blending external sounds exceeding a minimum audio loudness threshold into the rhythm and audio cues of the therapy. An example of an external sound is a car horn or an emergency vehicle siren, which synchronously and automatically interrupts normal auditory stimulation, thus alerting the listener to potential danger. To perform this and other functions, the listening device can have an additional microphone and digital signal processing dedicated to performing this task.

[0244] In another embodiment, the ANR system implementing AA can be configured to combine aspects of AA and spatial perception manipulation by aligning rhythmic auditory cues with the patient's "affected side" during a walking therapy session. For example, if the patient's right side requires greater treatment, the audio cue content can be spatially aligned with the right side for emphasis. Exemplary systems and methods for neurorehabilitation using side-specific rhythmic auditory stimulation techniques are disclosed in co-pending and co-assigned U.S. Patent Application No. 62 / 934,457, filed November 12, 2019, entitled "SYSTEM SANDMETHODS FORNEUROLOGIC REHABILITATION," the entire contents of which are incorporated herein by reference as if each were set forth herein in its entirety.

[0245] In another embodiment, an ANR system implementing AA can be configured to provide unique auditory cues to increase spatial awareness of head position during gait training, encouraging users to keep their heads up, in the midline, and with their eyes forward, improving balance and spatial awareness during gait training or other CTA experiences.

[0246] In another embodiment, the ANR system implementing AA can be configured to provide binaural beat sounds and correlate them with human physiology (e.g., respiratory rate, EEG activity, and heart rate) to improve cognition and enhance memory. The ANR system can be configured to provide binaural beat audio signal inputs complementary to the RAS signal input. The real-time entrainment and quality of gait measurements performed by the system will also be supplemented by physiological measurements. For example, a system configured for binaural beat audio uses differential frequency signals from the left and right ear outputs, with a difference of 40 Hz, the “gamma” frequency of neural oscillations. These frequencies can reduce the accumulation of amyloid protein in Alzheimer’s patients and help improve cognitive flexibility. By delivering such audio signals to the user via the AA device while the user performs RAS gait training, a second type of neural entrainment can be achieved simultaneously with biomechanical RAS entrainment. The network hypothesis of brain activation suggests that both walking and cognition are affected. Therefore, this auditory sensory stimulation entrains neural oscillations in the brain, while rhythmic auditory stimulation entrains the motor system.

[0247] In another embodiment, the ANR system implementing AA can be configured to provide a phase-coherent sound field (e.g., correct audio-spatial perspective) when the patient rotates their head or changes their posture. The sound field is a fictional three-dimensional image created by stereo speakers or headphones. It allows the listener to accurately hear the location of the sound source. An example of manipulating the sound field during therapy is keeping the virtual coach's voice "in front" of the patient, even if their head may turn to one side. This feature helps avoid disorientation, thus creating a more stable, predictable, and safe audio experience during therapy. This feature can be combined with... Figure 34 It combines AR virtual coaches / therapists in front of people. It can also be combined with knowledge of the course or direction a person needs to take in the real world.

[0248] In another embodiment, the ANR system can be configured to combine AA with augmented reality (AR) in such a way that virtual sound effects (e.g., encouragement and footsteps of the crowd) create a coherent sound field in response to the patient's visual gaze as the patient synchronizes with a virtual crowd. The audio can also generate a sense of distance or proximity to objects. Changing spatial location or loudness in this way can also be combined with AR and 3D imagery as target objects.

[0249] In another embodiment, the ANR system can be configured to combine AA with augmented reality (AR) in a way that creates virtual instrument therapy. Musical instruments such as bells, drums, pianos, and guitars can be commonly used training tools for patients. By creating digital models of these instruments and providing AA feedback during interaction, patients can gain an immersive experience and feel as if they are actually using the instruments. This can be modified based on difficulty to help patients progress over time and show improvement. Examples of modifications could include adding more keys to a piano or more strings to a guitar. In addition to virtual instruments, virtual sheet music or musical notation can be displayed in real time as the patient plays an instrument (virtual or real). Other examples can be combined with... Figure 19 The concepts discussed are combined, where the connected hardware can be replaced by AR. Similar logic can be used for interventions in other recordings.

[0250] In another embodiment, the ANR system can be configured to combine telepresence with analogue (AA) to provide therapists with a spatially accurate audio experience. Audio can also be generated to create a perception of distance or proximity to an object. By altering the spatial location or loudness of the AA and utilizing AR models, the system can be used to more effectively determine whether patients are meeting goals associated with playing virtual devices and to provide them with a more accurate and specific experience.

[0251] According to one or more embodiments, an ANR system can implement a type of AA called a Rhythm Stimulation Engine (RSE). A RSE is a customized rhythmic auditory stimulus that embodies the principle of entrainment to drive therapeutic effects while generating original and customized auditory rhythmic content for the patient. For certain disease states, such as Parkinson's disease, having a constant rhythmic "background music" in the patient's environment is also beneficial. The RSE can be configured to perform this continuous background rhythmic neural stimulation without access to pre-recorded music. In one example, the ANR system can be configured to combine the RSE and AR to implement AA, creating a fully synchronized feedback state between input biostatistics, external audio input from the therapeutic environment, and the generated rhythmic content, AR, and AA outputs. In another example, the system can be configured to interactively adjust the tempo of the rhythmic audio content generated by the RSE based on the patient's walking pace. In yet another example, the tempo and time signature of the rhythmic audio content generated by the RSE can be interactively adjusted by the entrainment precision and beat factor of the patient user (e.g., a user using a cane or assistive device). In another example, RSE can provide neural stimulation, which, combined with assistive technologies such as exoskeletons, exoskeletons, and / or FES devices, enhances the effectiveness of gait therapy. In yet another example, RSE can generate rhythmic audio content from a stored library of traditional dance rhythm templates, extending therapy to the patient's upper body and limbs. This can be extended to integration with the aforementioned AR technologies (e.g., a dancing crowd or a virtual dance floor). In yet another example, machine learning techniques such as self-learning AI and / or rule-based systems can generate real-time easing rhythms via inertial motion unit (IMU) input, which reports the quality of gait and gait parameters, such as symmetry and gait cycle time variability. Using unsupervised ML clustering or decision tree models, various gait patterns can serve as input to a music generation system.

[0252] According to one or more embodiments, an ANR system can achieve a type of AA, namely sonication, which means applying varying amounts of signal distortion to the music content based on the patient's distance from the target object. The degree and type of sonication help to push the patient into a corrected state. New combinations of sonication and banding can provide a feedforward mechanism for auditory-motor synchronization through banding, while simultaneously providing a feedback mechanism through distortion of the music content using some other biomechanical or physiological parameters that can be adjusted by the individual. For example, adding signal distortion to the music signal while simultaneously increasing the volume of rhythmic cues may be more effective when combined than using either method alone.

[0253] According to one or more implementation schemes, the ANR system can be combined with CTA administered via neurotoxin injection, as shown below. CTA can apply the entrainment principle to improve motor function, such as gait. Neurotoxin injections can also improve gait by targeting muscle spasms. These injections require 2–4 days to take effect and last for 90 days (e.g., the effective period). The dosage of CTA used for the entrainment principle (e.g., the setting of one or more parameters of the CTA) can be targeted to the effectiveness curve of the neurotoxin injection, where the training intensity is lower for a period before the injection takes effect and increases during the effective period.

[0254] According to one or more embodiments, an ANR system can be configured to calculate entrainment parameters using synchronization of heart rate or respiratory rate with music content, rather than biomechanical motion parameters. An example use case is for individuals suffering from various forms of dementia, Alzheimer's disease, bipolar disorder, schizophrenia, and other anxiety disorders. In this use case, baseline parameters can be determined via heart rate or respiratory rate. Entrainment, or phase entrainment, can be determined by comparing music content with heart rate or respiration. Furthermore, goals can be set to reduce anxiety levels to improve the quality of life for these individuals.

[0255] As can be seen from the above discussion, according to one or more embodiments of this disclosure, a system for enhancing patient neurological rehabilitation may include one or more of the following:

[0256] A computing system for having one or more physical processors configured with software modules including machine-readable instructions. The software modules may include a 3DAR modeling module, which, when executed by the processor, configures the processor to generate and present augmented reality visual and audio content to the patient during a treatment session. The content includes visual elements moving in a prescribed spatial and temporal sequence and rhythmic audio elements output at a beat speed.

[0257] The computing system also includes an input interface that communicates with the processor to receive input, which includes the patient's timestamped biomechanical data relating to movements performed by the patient in response to AR visual and audio content and using physical parameters measured using one or more sensors associated with the patient.

[0258] The software module also includes a critical thinking algorithm that configures the processor to analyze timestamped biomechanical data to determine the spatial and temporal relationships of the patient's movement relative to visual and audio elements, and to determine the patient's entrainment level relative to target physiological parameters. Furthermore, the 3DAR modeling module configures the processor to dynamically adjust the augmented reality visual and audio content output to the patient based on the entrainment level determined relative to the target parameters.

[0259] The aforementioned systems, devices, methods, processes, etc., can be implemented using hardware, software, or any combination thereof suitable for the application. Hardware may include general-purpose computers and / or special-purpose computing devices. This includes implementation in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices or processing circuits, as well as internal and / or external memory. This may also or alternatively include one or more application-specific integrated circuits, programmable gate arrays, programmable array logic components, or any other device that can be configured to process electronic signals. It will also be appreciated that implementation of the aforementioned processes or devices may include computer-executable code created using a structured programming language such as C, an object-oriented programming language such as C++, or any other high- or low-level programming language (including assembly language, hardware description languages, and database programming languages ​​and techniques), which can be stored, compiled, or interpreted to run on one of the aforementioned devices, and processors, heterogeneous combinations of processor architectures, or combinations of different hardware and software. On the other hand, the methods may be embodied in a system that performs its steps and may be distributed across devices in various ways. Simultaneously, processing may be distributed across devices such as the various systems described above, or all functions may be integrated into dedicated, standalone devices or other hardware. On the other hand, the means for performing the steps associated with the above-described process may include any of the aforementioned hardware and / or software. All such permutations and combinations are intended to fall within the scope of this disclosure.

[0260] The embodiments disclosed herein may include computer program products comprising computer-executable code or computer-usable code that, when executed on one or more computing devices, performs any and / or all of its steps. The code may be stored in a non-transitory manner in computer memory, which may be memory from which the program executes (e.g., random access memory associated with a processor), or storage devices such as disk drives, flash memory, or any other optical, electromagnetic, magnetic, infrared, or other devices or combinations thereof. In another aspect, any of the systems and methods described above may be embodied in any suitable transmission or propagation medium carrying the computer-executable code and / or any of its inputs or outputs.

[0261] It should be understood that the above-described devices, systems, and methods are illustrated by way of example and not limitation. Without explicit instructions to the contrary, the disclosed steps may be modified, supplemented, omitted, and / or rearranged without departing from the scope of this disclosure. Many variations, additions, omissions, and other modifications will be apparent to those skilled in the art. Furthermore, the order or presentation of the method steps in the above description and figures is not intended to require that the steps be performed in that order, unless explicitly required or otherwise required by the context.

[0262] The method steps of the embodiments described herein are intended to include any suitable methods to enable these method steps to be performed and to comply with the patentability of the following claims, unless a different meaning is explicitly provided or the context otherwise expressly requires. Thus, for example, performing step X includes any suitable method for causing another party, such as a remote user, a remote processing resource (e.g., a server or cloud computer), or a machine, to perform step X. Similarly, performing steps X, Y, and Z may include any method for directing or controlling any combination of these other individuals or resources to perform steps X, Y, and Z to obtain the benefits of these steps. Therefore, the method steps of the embodiments described herein are intended to include any suitable method for causing one or more other parties or entities to perform steps in accordance with the patentability of the following claims, unless a different meaning is explicitly provided or the context otherwise expressly requires. These parties or entities need not be directed or controlled by any other party or entity, nor need they be located in a particular jurisdiction.

[0263] It should also be understood that the above methods are provided by way of example. Without explicit instructions to the contrary, the disclosed steps may be modified, supplemented, omitted, and / or rearranged without departing from the scope of this disclosure.

[0264] It should be understood that the methods and systems described above are illustrated by way of example and not limitation. Many variations, additions, omissions, and other modifications will be apparent to those skilled in the art. Furthermore, the order or presentation of the method steps in the above description and figures is not intended to require a specific order of performance unless a particular order is explicitly required or otherwise required by the context. Therefore, while specific embodiments have been shown and described, it will be apparent to those skilled in the art that various changes and modifications to the form and details therein can be made without departing from the spirit and scope of this disclosure.

Claims

1. An enhanced neurorehabilitation system for patients, comprising: A computing system having a processor configured with software modules containing machine-readable instructions stored in a non-transitory storage medium. The software module includes: An AA / AR modeling module, when executed by the processor, configures the processor to generate augmented reality (AR) visual content and rhythmic auditory stimulation (RAS) for output to a patient during a treatment session, wherein the RAS includes a beat signal output at a beat speed, and wherein the AR visual content includes visual elements moving in a prescribed spatial and temporal sequence based on the beat speed. An input interface, communicating with the processor, is used to receive real-time patient data, the real-time patient data including timestamped biomechanical data of the patient related to repetitive movements performed by the patient in a timely manner with the AR visual content and RAS, wherein the biomechanical data is measured using sensors associated with the patient; and The software module further includes: A Clinical Thinking Algorithm (CTA) module is configured to configure the processor to analyze the timestamped biomechanical data to determine the temporal relationship between the patient's repetitive movements and the visual elements, and the temporal relationship between the beat signal output at the beat rate, to determine the entrainment level; and The AA / AR modeling module further configures the processor to dynamically adjust to the patient's AR vision and RAS output synchronously based on the determined entrainment level. The dynamic adjustment of the AR visual content includes: adjusting the beat rate of the RAS according to the entrainment level, and adjusting the prescribed spatial and temporal sequence of the visual elements synchronously with the adjusted beat rate.

2. The system according to claim 1, in, Each beat signal is output at its corresponding output time; The entrainment level is determined based on the timing of the repetitive motion relative to the corresponding output time of the beat signal.

3. The system according to claim 2, wherein, The CTA module configures the processor to determine the entrainment level in the following manner: The timestamped biomechanical data were analyzed to identify the corresponding times for repetitive movements. Measuring the time relationship between the corresponding times of one or more repetitive motions and the corresponding output times of one or more associated beat signals. The entrainment potential is calculated based on the time relationship measured for one or more of the corresponding repetitive motions; as well as The processor is configured to dynamically adjust one or more of the AR vision and RAS outputs based on the entrained potential.

4. The system according to claim 1, in, The CTA module is further configured to determine, based on one or more of the physiological and biomechanical data measured for the patient, whether one or more of the biomechanical and physiological data meet the training target parameters; The AA / AR modeling module further configures the processor to dynamically adjust the patient's AR visual content and RAS output in response to the failure to meet the training target parameters by adjusting the beat speed of the RAS according to the failure to meet the training target parameters and adjusting the prescribed spatial and temporal sequence of the visual elements in sync with the adjusted beat speed.

5. The system according to claim 4, wherein, The training target parameter is the rehabilitation outcome.

6. The system according to claim 1, further comprising: An AR video output device is configured to present the AR visual content to the patient. An audio output device configured to output the RAS to the patient; as well as The sensor, which is associated with the patient and configured to measure the patient's timestamp biomechanical data, includes an inertial measurement unit (IMU) device.

7. The system of claim 4, further comprising: A sensor, associated with the patient and configured to measure the patient's physiological data, wherein the training target parameter is a physiological parameter, and The physiological parameters mentioned are one or more of heart rate, blood oxygen, respiratory rate, VO2, and EEG activity.

8. The system according to claim 1, wherein, The AR visual content includes visual scenes, which include one or more of the following: A virtual treadmill, animated to appear as if the top surface of the treadmill were approaching the patient at a rate corresponding to the beat speed, and Multiple footprints are superimposed on the top surface of the virtual treadmill, wherein the footprints are arranged in space and appear to approach the patient at a rate corresponding to the beat speed, and An animated character that performs repetitive actions at a rate corresponding to the beat speed.

9. The system according to claim 8, wherein, Modifying the AR visual content in sync with the adjusted beat speed includes one or more of the following: changing the spacing of the footprints according to the change in beat speed, changing the rate at which the animator performs repetitive actions, changing the rate at which the virtual top surface of the treadmill appears close to the patient, and changing the rate at which virtual obstacles or scene disturbances appear in front of the patient.

Images