Progressively adaptive generative "smart" instrument for personalized human motion-based musical compositions and sonification

Abstract

The progressively adaptive generative "smart" instrument for personalized human motion-based musical compositions and sonification of the present invention is an algorithm / apparatus / device for neurorehabilitation, therapy, and performance using human motion to generate music or data sonification by integrating computer vision (CV) with adaptive musical instrumentation. The present invention provides a real-time generative music engine and framework that integrates digital symbolic representations of music, audio inputs or files, text or voice-prompts, audio inputs or files, and / or musical artifacts. Further, the present invention includes access to a database that stores and catalogs the elements which can be used to guide or generate a personalized musical composition or sonified output, integrating CV and the generative framework, developing a modular "development" style framework, developing progressively adaptive mappings for personalized / customized user interactions with motion-based musical composition and sonified motion, and correlating music and motion data in a digital analytics platform.

Description

TITLE OF THE INVENTION: Progressively Adaptive Generative “Smart” Instrument forPersonalized Human Motion-Based Musical Compositions and SonificationBACKGROUND OF THE INVENTIONTechnical Field

[0001] This invention relates generally to an adaptive, generative smart musical instrument for personalized human motion-based musical compositions and sonification. This invention relates more particularly to an algorithm / apparatus / device for neurorehabilitation and / or therapy / treatment using human motion tracked via computer vision (CV) with adaptive generative music algorithms to sonify motion and / or mediate music generation, which may collect data related to the user’s interactions with the system to personalize the generative musical or sonified output and user interaction mappings to assist in targeted user health outcomes.

[0002] This invention relates generally to algorithms, apparatuses, and devices for neurorehabilitation using music. This invention relates more particularly to neurorehabilitation using generative music mediated by CV, digital symbolic representations of music, audio inputs or files, text and / or voice commands of musical characteristics, user-selected music genre sets, and progressively adaptive mappings.Background Art

[0003] The novelty of the present invention is at least six-fold: integrating CV with adaptive musical instrumentation, providing a real-time generative music engine and framework that integrates digital symbolic representations of music, audio inputs or files, text or voice-prompts,audio inputs or files, and / or musical artifacts to guide a personalized musical or sonified output, integrating CV and the generative framework, developing a modular “development” style framework, developing progressively adaptive mappings for personalized / customized user interactions with motion-based musical composition and sonified motion, and correlating music and motion data in a digital analytics platform.

[0004] First is integrating CV into an adaptive generative smart musical instrument. There are few, if any, commercially available software or hardware musical instruments that leverage CV for real-time control of adaptive generative musical parameters. Some CV-based projects exist that allow musical control but are in their infancy, or are open-source, and are limited in their ability to individualize the CV mappings to a broad range of user needs and musical aesthetics. Other adaptive CV-to-generative music implementations can be found in research or performance projects, but are again limited in their adaptability, musical range, and scope - additionally, none are currently commercially available. The use of nonphysical-based sensor control of any parameter is rare in commercial hardware musical instruments but does exist, however they offer a limited number of motion-to-music controllers and are not adaptive, with no adaptive control over aspects of generative musical elements. Alternatively, CV can provide tracking data for hundreds of tracking points (e.g. wrist joints, etc.) on the human body which is unprecedented in these current commercial iterations of sensor-based music controllers, to allow the potential for an expansive range of fluid, hands-free control over motion-based generative music parameters.

[0005] The present invention includes CV machine learning models which are able to identify and classify poses, gestures, facial expressions, relationships of bodily movement, and evenscenery such as forests, office buildings, bedrooms, and beaches. These characteristics may be integrated into the music generation, movement sonification mappings, and / or visual feedback.

[0006] Second is providing a real-time generative music framework that integrates and / or responds to digital symbolic representation of music (i.e. digital lead sheet or non-westem equivalent), audio inputs or files, text or voice-prompts, audio inputs or files, and / or musical artifacts, genre set, user-selected musical characteristics, harmonic changes, melody input, and rhythmic elements for personalized musical and sonified outputs. The music generation portion of the present invention adds another layer of novelty to the instrument. In general, the generative music algorithm will combine machine “learned” concepts aggregated from sampled professional musicians and musical compositions, as well as traditional rules from music theory to generate realistic, multi-part compositions in real-time that can morph in complexity levels and adapt to the user. The generative music will be based primarily on an input of digital symbolic representations of music, which provide minimal harmonic, melodic, and rhythmic information of music or a specific song, text and / or voice prompts of musical characteristics, and a user-selected musical genre set. This “lead sheet” approach may provide only a simplistic “nuts-and-bolts” representation of the song (simplified harmony, melody, rhythm) which may then be constantly re-interpreted in real-time by the user’s motion, and ultimately guided by rhythmic, harmonic, and melodic patterns that are consistent with the selected genre(s) and musical characteristics. This approach gives vast amounts of creative potential for musical expression, as the lead sheet provides only minimalistic information about the song harmony, melody, and rhythm, allowing motion to mediate musical variations that are consistent with stochastic and rule-based representations of genre-specific musical and rhythmic patterns anduser preferences in musical aesthetics. This generative musical framework is also novel in that it may be augmented or supplemented by audio files or inputs (such as voice clones) and text or voice prompts of musical characteristics.

[0007] Third is integrating CV and the generative framework. The integration of CV and a generative framework would present transformative concepts for applications in and beyond music and music-based therapeutic modalities and approaches. In particular, the ability to morph the complexity of generated melodies, harmonies, rhythms, timbres, and / or instrumentation set selection in real-time, with generated music guided by a lead sheet, selected musical characteristics, and genre input, while also responding and adapting to user body motion (provided by CV) would be the first of its kind. It is also worth considering that most hardware instruments have only simplistic generative capabilities (e.g., arpeggiator and sequencer) that are triggered with traditional button presses. Moreover, software-based generative algorithms are not typically designed to function in real-time and are usually presented in application form, not as an integrated, adaptive musical instrument. In general, the present invention presents new individual concepts (adaptive, real-time CV-based musical control) for functional musical instruments that are novel individually as well as when integrated with other technologies (generative music, digital therapeutic interventions).

[0008] Fourth is developing progressively adaptive mappings for personalized / customized user interactions with motion-based musical compositions and sonified motion. This framework allows machine learning and artificial intelligence (Al) to leverage user data and / or a database of training data collected from multiple users or other relevant populations to adjust the CV mappings, generative music and sonified motion mappings, visualizer mappings, and / orcombinations of the above to meet the needs and objectives of the user and their target use case implementation. In a rehabilitation context, the term “progressive adaptation” describes a gradual increase in the demand that is placed on an individual’s physical or cognitive abilities in response to the individual becoming adapted to existing demands. Progressive adaptation is used to forestall a plateauing of an individual’s abilities during rehab. Many progressive adaptation techniques already exist, but the “smart” technique that the described instrument will utilize is one-of-a-kind. This innovation allows the instrument to use proprietary analysis of a user’s performance metrics in real-time to continuously select the right mix of ‘demand dimensions’ (frequency, number of reps, speed, range, etc.) to increase so that the user’s abilities and performance outputs increase continuously. In contrast, standard techniques for progressive adaptation typically increase demand in a disjointed, periodic manner while focusing on just one ‘demand dimension’ at a time, which often results in small but noticeable dips in an individual’s performance output.

[0009] Fifth is developing a modular “development” style framework for “drop-in” implementation in existing or developing di ital / augmented reality platforms (i.e. games, puzzles, exercises, therapeutic interventions). The described adaptive, smart instrumentation introduces promising potential for a development framework for 3rd party use. Similar to app development on smart-phone hardware (e.g. games), the described instrument engine could provide near limitless possibilities for CV-mediated musical compositions and sonified motion for developers. A basic software development kit (SDK) could be implemented that abstracts the main function calls and provides third party software developers with a tool to create their ownsoftware and implement novel applications of motion-based musical compositions or sonifi cation using either the described CV mapping engine, generative engine, or both.

[0010] Sixth is correlating music and motion data in a digital analytics platform. A key feature of the present invention’s integration with digital therapeutics will be its ability to collect, organize, and analyze user data (i.e. health data, movement data, user-interactions with generative music) and to leverage this data to progressively adapt its mappings to facilitate the user’s therapeutic progress using machine learning or Al. To accomplish this, it must record data on real-time user events (i.e. musical elements time-stamped vs. CV motion data), identify musical characteristics with strong associations to positive user outcomes, integrate and interpret multiple forms of user health data (i.e. Autonomic Nervous System (ANS) data, heart rate, medical imaging, electronic brain signals, 02 levels) and provide organized displays of user progress: to improve treatment efficiency. Ultimately this data will be leveraged for use in a bioinformatics system that will correlate user health and motion data with musical events and characteristics to elucidate stochastic relationships that are relevant to the impact of music on user health outcomes.Additionally this system will allow medical practitioners, clinicians, and therapists to: (i) record real-time user / patient events during clinical sessions and create informational notes about the session when it ends, and (ii) access analytics which identify musical events that are strongly associated with positive user / patient outcomes. The totality of this information will help medical practitioners, clinicians, therapists, and care providers / partners design therapeutic sessions that systematically improve and accelerate user / patient progress. This information is also anticipated to be central to hastening the adoption of data-driven, personalized music-based treatments into mainstream medical science. Further, these analytics will allow the harmonic, melodic, rhythmic,and timbral characteristics and constraints of the adaptive generative smart instrument, motion- to-music mappings, and integrated neurorehabilitation / therapeutic interventions to be progressively modified by the software and / or practitioner to afford the user / patient higher levels of individualized creative musical freedom (i.e. improvisation) as neurorehabilitation / therapeutic goals are achieved. This feature is intended to empower users with an interactive music creation experience that is increasingly expressive and personalized in parallel with their neurorehabilitation / therapeutic progress (conceptually analogous to removing the bumpers from bumper bowling as the player becomes more proficient at the game).

[0011] In light of the foregoing prior art, there is a need for a progressively adaptive generative “smart” musical instrument for personalized motion-based musical composition and sonifi cation, an adaptive human motion generative musical instrument using CV, a human motion-mediated generative musical instrument using CV to broaden access to individualized music-based approaches for neurorehabilitation using human motion to generate personalized music and sonified outputs, and a human motion generative musical instrument using CV to facilitate individualized music-based therapeutics to better provide improved user health outcomes.BRIEF SUMMARY OF THE INVENTION

[0012] The invention is a CV-based software musical instrument that maps human motion to realistic, real-time generated musical compositions and sonified outputs based on digital symbolic representations of music that contain minimal information about harmony, melody, rhythm, and selected musical genre. The main application is intended for therapeutic and / or neurorehabilitative treatments in clinical and at-home environments (e.g., music therapy) butcould be marketed towards the general public and professional musician as well. The human motion input is tracked with CV which is advantageous as it can capture positions, velocities, accelerations, and relationships of motion of key tracking points on the human body without the use of sets of expensive or cumbersome motion sensors that would otherwise provide the same information. This motion is then mapped to musical parameters that control the real-time generation & texture of the musical performance itself, including traditional parameters such as music transport control (e.g., speed of playback, etc.), effects depths, synthesizer parameters, as well as novel “smart” parameters that control the complexity of the generated stream of music notes (e.g., harmonic voicing density, rhythmic feel, “level” of musicianship, etc.). The latter will be based on sampled & analyzed musical data aggregated from professional musicians, musical compositions, as well as existing rules from music theory converted into computer code.

[0013] According to a first aspect of the invention there is a generative instrument for playing music from a digital symbolic representation of a song, an audio input or file, a text or voiceprompt, or a musical artifact used for creating a musical output comprising a CV system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a real-time performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech generation, a musical speech generation, a singing performance generation, and / or amusical instrumentation set selection which is connected to a real-time user CV control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of their body are being tracked and a real-time visual feedback that shows said user how their motions and / or gestures are being interpreted and used to alter said musical output.

[0014] According to a second aspect of the invention there is a generative instrument further comprising a knob / slider based control configured to operate said generative instrument.

[0015] According to a third aspect of the invention there is a generative instrument further configured to capture a position, a velocity, an acceleration, and / or relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position. Optionally, wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output. And optionally, wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

[0016] According to a fourth aspect of the invention there is a generative instrument further configured to function either in real-time interactively or as a non-interactive automatic musicgenerator (with parameters specified at and / or prior to run-time) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said musical instrument and / or as specified by the user prior to interacting with said musical instrument.

[0017] According to a fifth aspect of the invention there is a generative instrument further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity or a configuration of a generated stream of music notes.

[0018] According to a sixth aspect of the invention there is a generative instrument wherein said digital symbolic representation of a song or musical composition comprises of a harmonic structure and a melody of said song (or an equivalent non-western digital representation of a musical composition), and may be modified by text or voice-prompts of musical characteristics and / or a user-selected musical genre set or a non-western equivalent.

[0019] According to a seventh aspect of the invention there is a generative instrument wherein said digital symbolic representation of a song, or text or voice prompt, may only provide minimalistic information about a harmony, a rhythm, a melody, an instrumentation set, timbral characteristics, musical characteristics, and / or a genre set, allowing said human motion to generate and mediate musical variations consistent with stochastic and rule-based representations of genre-specific musical and rhythmic patterns and user selected musical preferences.

[0020] According to an eighth aspect of the invention there is a generative instrument wherein said digital symbolic representation of a musical composition / song may be stored and cataloguedby titles, musical characteristics, and / or other characteristics so that users can easily customize their musical interactions and creative engagement through access to a bank of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced.

[0021] According to a ninth aspect of the invention there is a generative instrument further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of a user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output.

[0022] According to a tenth aspect of the invention there is a generative instrument wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of musical play.

[0023] According to an eleventh aspect of the invention there is a generative instrument further configured to map to speech, musical speech, singing performance(s), and / or sound effects in a deterministic manner, wherein said speech, musical speech, singing performance, and / or sound effects are determined by the output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth,a set of dynamics, a synthesizer parameter, a smart parameter, and / or and timbre transfers to give an expansive and dynamic range of audible outputs that may be customized by said user.

[0024] According to a twelfth aspect of the invention there is a generative instrument further configured to collect and record at least one data element based on at least one real-time interaction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user performance outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user performance outcome by analyzing any / all data accessible by said generative instrument, to suggest new or modified interactions with said generative instrument that the user should engage in to enable an improvement in said user performance outcome, and to provide an organized display of a user performance progress.

[0025] According to a thirteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music for engaging in digital music-based neurorehabilitative interventions from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voice-prompt, or a musical artifact creating a musical output comprising a computer-vision system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a real-time performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbregeneration, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of their body are being tracked and / or a real-time visual feedback that shows said user how their motions and / or gestures are being interpreted and used to alter said musical output.

[0026] According to a fourteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to integrate with a digital neurorehabilitative intervention set designed to stimulate a neurochemical environment in the user’s brain that facilitates mechanisms of neuroplasticity, to analyze a user health data and / or a neurological health data, to learn and remember changes in said user’s health data and / or neurological health data and engagement with said neurorehabilitative music that reflect changes in the user’s neurological health outcomes, and to progressively adapt said musical output and the user’s engagement with said neurorehabilitative music and a music-based neurorehabilitative intervention set with machine learning to facilitate or target a therapeutic progress of said user using any combination of a physical interaction, an emotional interaction, a cognitive interaction, a sensory interaction, a social interaction, and / or a physiological interaction. Optionally, wherein said physical interaction, said emotional interaction, said cognitive interaction, said sensory interaction, said social interaction, and / or said physiological interaction are combined with a medical image data to develop statistical correlations of the health and / or neurological health impact of music.

[0027] According to a fifteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music further comprising a knob / slider based control configured to operate said generative instrument.

[0028] According to a sixteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to capture a position, a velocity, an acceleration, and / or relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position. Optionally, wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output. And optionally, wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

[0029] According to a seventeenth aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to function in either real-time interactively or as a non-interactive automatic music generator (with all parameters specified at run-time) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of saidmusical instrument and / or as specified by the user prior to interacting with said musical instrument.

[0030] According to an eighteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity or a configuration of a generated stream of music notes.

[0031] According to a nineteenth aspect of the invention there is a generative instrument for playing neurorehabilitative music wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of musical play.

[0032] According to a twentieth aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to map to speech, musical speech, singing performance(s), and / or sound effects in a deterministic manner, wherein said speech, musical speech, singing performance(s), and / or sound effects are determined by the output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or and timbre transfers to give a dynamic range of audible outputs that may be customized by said user.

[0033] According to a twenty -first aspect of the invention there is a generative instrument for playing neurorehabilitative music further configured to collect and record at least one data element based on at least one real-time interaction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user neurorehabilitation outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user neurorehabilitation outcome by analyzing any / all data accessible by said generative instrument, to suggest new or modified interactions with said generative instrument that the user should engage in to enable an improvement in said user neurorehabilitation outcome, and to provide an organized display of a user neurorehabilitation progress.

[0034] According to a twenty-second aspect of the invention there is a generative bioinformatics treatment / therapy instrument for playing music from a digital symbolic representation of a musical composition / song, an audio input or file, a text or voice-prompt, or a musical artifact creating a musical output comprising a computer-vision system configured to capture a human motion, a data-driven digital bioinformatics system for evaluating a user / patient health and targeting said user / patient health outcomes, a mapping system having a data acquisition system with a plurality of data elements, one of said plurality of data elements being said patient outcome configured to translate said human motion into a plurality of musical parameters adjusted by said patient outcome, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a realtime performance of a realistic harmony / chord voicing, a voice leading, an accompanimentgeneration, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of said user configured so said user can see which parts of their body are being tracked and / or a real-time visual feedback that shows said user how their motions and / or gestures are being interpreted and used to alter said musical output wherein said mapping system is configured to adjust said plurality of musical parameters based on said plurality of data elements.

[0035] According to a twenty -third aspect of the invention there is a generative bioinformatics treatment / therapy instrument further comprising a knob / slider based control configured to operate said generative instrument.

[0036] According to a twenty-fourth aspect of the invention there is a generative bioinformatics treatment / therapy instrument further configured to collect and record said plurality of data elements based on a plurality of real-time user / patient events, to identify and remember a plurality of musical characteristics one of said plurality of musical characteristics being transport controls with strong associations to a positive user / patient health outcome, to progressively adapt said generative treatment / therapeutic instrument to identify a change in said user’s / patienf s health outcomes by analyzing all relevant health and rehabilitation data accessible by said generative treatment / therapy instrument, to facilitate targeted improvements in said user’s / patient’s health, and to provide an organized display of user / patient health / therapeutic / rehabilitative progress.

[0037] According to a twenty-fifth aspect of the invention there is a generative bioinformatics treatment / therapy instrument further configured to capture a position, a velocity, an acceleration, and relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position and / or a hand position. Optionally, wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output. And optionally, wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

[0038] According to a twenty-sixth aspect of the invention there is a generative bioinformatics treatment / therapy instrument further configured to either function in a real-time interactive mode or in a non-interactive automatic music generator mode (with all parameters specified at runtime) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said musical instrument and / or as specified by the user prior to interacting with said musical instrument.

[0039] According to a twenty-seventh aspect of the invention there is a generative bioinformatics treatment / therapy instrument further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamic parameters, a synthesizer parameter, and / or a smart parameter that control a complexity or a configuration of a generated stream of music notes.

[0040] According to a twenty-eighth aspect of the invention there is a generative bioinformatics treatment / therapy instrument wherein said generative instrument is configured to be progressively adaptive, changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy.

[0041] According to a twenty -ninth aspect of the invention, there is a generative instrument for playing neurorehabilitative music further configured to map to speech, musical speech, singing performance(s), and / or sound effects in a deterministic manner, wherein said speech, musical speech, singing performance(s), and / or sound effects are determined by the output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or and timbre transfers to give a dynamic range of audible outputs that may be customized by said user.

[0042] According to a thirtieth aspect of the invention, there is a generative bioinformatics treatment / therapy instrument for playing neurorehabilitative music further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality ofmetrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output. This sensor system can function in conjunction with, and / or separately from said CV-based motion sensing system,. The sensor system can also be used to improve the motion detection itself via higher resolution capture or provide new types of information that are difficult or impossible to capture with CV alone.

[0043] According to a thirty-first aspect of the invention, there is a generative bioinformatics treatment / therapy instrument for playing neurorehabilitative music wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

[0044] Advantages of the generative instrument of the present invention include:

[0045] Infinite variability to musical output;

[0046] Advantageous to traditional therapeutic (both music-based and non-musical) approaches in that it harnesses the known benefits of music to stabilize and / or improve executive functions and memory, neuromotor abilities, sensory perception, emotional self-regulation, behavior management, and socialization abilities;

[0047] Active participation in the music creation process by users with their bodily movements to maximize distributed cross-hemispheral neural activity in the user and stimulate a neurochemical environment and neurological mechanism(s) that increase(s) their brain’s capacity for neuroplasticity.

[0048] Recording data on real-time patient events, identifying musical characteristics with strong associations with positive patient outcomes; and

[0049] Providing organized displays of patient progress to maximize the efficiency and effectiveness of a user’s music-based therapy or treatment.

[0050] The invention will now be described, by way of example only, with reference to the accompanying drawings in which:BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0001] Figure 1 is a flow chart of the system operations according to the present invention. Specifically, this figure shows a possible software design route (but not necessarily the only possible design) that illustrates the required inputs to the software, the main generative engine functionality, the CV integration, as well as the overall tone generation.

[0052] Figure 2 is a flow chart of the system operations according to the present invention.Specifically this figure illustrates one possible conceptual technical design for the interactive visualizer.DETAILED DESCRIPTION OF THE INVENTION

[0053] The detailed embodiments of the present invention are disclosed herein. The disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. The details disclosed herein are not to be interpreted as limiting, but merely as the basis for the claims and as a basis for teaching one skilled in the art how to make and use the invention.

[0054] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etcetera, indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such a feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0055] Furthermore, it should be understood that spatial descriptions (e.g., “above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,” “vertical,” “horizontal,” etc.) used herein are for purposes of illustration only, and that practical implementations of the structures described herein can be spatially arranged in any orientation or manner.

[0056] Throughout this specification, the word “comprise,” or variations thereof such as “comprises” or “comprising,” will be understood to imply the inclusion of a stated element, integer or step, or group of elements integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.

[0057] Throughout this specification, the word “sonification” will be understood to imply the process of representing data or information audibly or as sound. This involves the conversion of non-auditory data, such as motion or numerical data or patterns into reproducible, audible signals, allowing for interpretation or data analysis through auditory rather than visual means. The same data streams that are used to generate musical outputs in the present invention could also be represented through objective means of sonification, which are likely to sound less musical. The musical and sonified outputs could be played together or as substitutes for one another depending on the use case scenario.

[0058] Throughout this specification, the phrase “generative engine” includes but is not limited to an electronic computer based system capable of creating a musical or sonified output from any sound input as manipulated by a set of controls and / or mapping functions.

[0059] Throughout this specification, the phrase “music transport” or variations thereof such as “music transport controls” includes but is not limited to play, stop, pause, and / or rewind. This is one of the few controls that could be best suited for physical buttons / keys because it serves as a failsafe in case an obvious failure of CV interpretation; for example, the user performs a paused hand motion that doesn’t get tracked properly and the song continues.

[0060] Throughout this specification, the word “dynamics,” or variations thereof such as “set of dynamics” includes but is not limited to in-process alterable control and content.

[0061] Throughout this specification, the word “offline” includes but is not limited to any play or use occurring that is not a live performance.

[0063] Throughout this specification, the phrase “smart parameter” includes but is not limited to any adjustable or editable control or element used in the music creation or sonification process and / or the mapping of a body movement or emotion.

[0064] Throughout this specification, the phrase “lead sheet” includes but is not limited to a digital symbolic representation of a song or musical artifact. Alternate terms would include fake sheets, guitar charts, “real book” charts, or scores. The lead sheet can be any other non- conventional representation of a song, such as a vocal input, any non-westernized forms of music representation, or a text / voice prompt of musical or auditory characteristics.

[0065] Stochastic and rule-based representations of music are derived from collecting and analyzing musical data from professional musicians and musical compositions. The stochastic relationships and musical rules that are derived will be based off the analysis of professional studio musicians and musical compositions.

[0066] Tracking points will be interpreted in real-time by custom software and classified into the following categories:

[0067] Gestures / poses / facial expressions. These could be individual hand gestures (i.e. thumbs up, finger pointing, snapping, a hand wave, etc.), dual hand gestures (i.e. hand clapping, air drumming, etc.), full body micro gestures (i.e. weight shifts, torso twists, hip / shoulder / knee actions, etc.), full body compound gestures (i.e. slide, box step, general dance steps, hands on hips, hands crossing chest, dual arm raise, etc.), facial expressions (i.e. smiling, frowning, yawning etc.), or combinations of the above. Such gestures / poses may serve as triggers whichalter the mode (i.e. the currently active algorithmic subroutine) of real-time music generation algorithm.

[0068] Real-time control parameters. When the program is in a particular mode as specified by the above classes of gestures / poses / facial expressions, other forms of computation on data-points can be used to directly control music generation parameters that are continuous in nature (i.e. a graduated value between 0 and 1, as opposed to a boolean 0 or 1 as in the above gesture triggers). For example, the angle of an arm or a hand can be one such parameter. Another may be the combined average velocity of all hand tracking points, taken over some durational window (a moving average).

[0069] The generative musical instrument of the present invention can be used for playing music from a music lead sheet comprising a generative instrument for playing music from a lead sheet comprising a computer-vision system configured to capture a human motion and a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being music transport control.

[0070] The characteristics of the present invention include but are not limited to: a CV-based software musical instrument; adaptive mappings of human motion to realistic (i.e. professional “sounding”), real-time generated musical compositions; human motion input tracked via CV; CV captured positions, velocities, and accelerations of key tracking points on the human body; tracked motion mapped to musical parameters that control the real-time generation & texture of the musical performance itself or data sonification parameters, including traditional parameters such as music transport control (e.g, speed of playback, etc), effects depth, synthesizerparameters, as well as new “smart” parameters that control the complexity of the generated stream of music notes (e.g., chord voicing density, rhythmic feel, “level” of musicianship, etc).

[0071] The generative music may be guided by or based on a “lead sheet” input, which provides only a simplistic representation of chords / harmony, melody, and rhythm of the song, and can be influenced by the user or software selection of musical features such as musical genre. This enables vast amounts of creative potential for the song, as the lead sheet only provides minimalistic information about the song harmony, melody, and rhythm. This allows motion to mediate musical variations that are consistent with stochastic and rule-based representations of genre-specific musical and rhythmic patterns intended for therapeutic treatments in clinical and at-home environments with the ability to analyze patient data, progressively adapt to facilitate the user’s therapeutic progress using machine learning, record data on real-time patient events (i.e. musical elements time-stamped vs. CV motion data), identify musical characteristics with strong associations to positive patient outcomes, and provide organized displays of patient progress to improve treatment efficiency. Note that versions of this platform could be marketed towards the general public and professional musician as well.

[0072] The present invention can be used for any neurorehabilitative purpose wherein the generative instrument for playing neurorehabilitative music from a lead sheet comprises a computer-vision system configured to capture a human motion and a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being music transport control.

[0073] Data acquisition includes not just collecting data on music and movement but also patient outcomes for any / all user / patient health conditions.

[0074] The present invention is a generative treatment / therapy instrument for playing music from a lead sheet comprising a computer-vision system configured to capture a human motion, and a patient outcome. Digital and analog metrics of patient health / function are the umbrella, and the device of the present invention can use used to augment any patient treatment mode, method, or means.

[0075] The present invention is a generative treatment / therapy instrument that can be further configured to collect and record said plurality of data elements based on a plurality of real-time patient events, to identify a plurality of musical characteristics with strong associations to positive patient outcomes, and to provide a plurality of organized displays of patient progress for the purpose of improving treatment efficiency.

[0076] The present invention includes a mapping system configured to translate said human motion into a plurality of musical parameters adjusted by said patient outcome, one of said plurality of musical parameters being music transport control, wherein said mapping system further comprises a data acquisition system having a plurality of data elements, one of said plurality of data elements being a patient outcome, wherein said mapping system is configured to adjust said plurality of musical parameters based on said plurality of data elements.

[0077] The present invention further enables physically intuitive motion-to-music “smart” mappings that support adaptive, individualized control over generated music using machine learned and Al learned structures and nuances of bodily movements in genre-specific, intragenre, and inter-genre related musical contexts. These smart, responsive mappings are a contrast to more deterministic movement mapping, made by pre-assigning movements onto selectgenerative musical elements (i.e. arpeggiations, melodic variations, etc.), that are inflexible and largely outside the user’s intuitive control.

[0078] Many musical movement styles appear to make logical (or physically intuitive) sense to a given genre of music when the movement is made in response to the music (i.e. classical music elicits fluid ballet-like movements, jazz music elicits syncopated movements). Given this consideration, our assumption is that physically intuitive musical movements (i.e. genre-specific, improvisational, or interpretive dance moves etc.) that are prompted by music can be represented via interdependent stochastic processes by virtue of natural, humanistic “cause-and-effecf ’ relationships between interdependent hierarchies of music and musical motions. These relationships can be translated into adaptive, individualized CV mappings which preserve musical intuition by reversing the typical direction of cause-and-effect (from music-to-motion to motion-to-music), so that the mapping supports movement that can generate physically intuitive, genre-related music, which may additionally be calibrated to the user’s capabilities.

[0079] One possible approach to implement such a smart mapping would leverage correlations between natural probability distributions of movement hierarchies and interdependencies that are meaningful to musical movement, which have been derived from collecting and analyzing movement data from professional dancers and choreographers. Stochastic and rule-based representations of micro movement sequences in time at each major landmark in the body may be characterized quantitatively using kinematic parameters (i.e. weight shifts, knee actions, etc ). Sequences of micro-movements derived from genre-specific, improvisational, or interpretive dance moves at each body landmark may be clustered using machine learning and then qualitatively labeled (i.e. box step, waltz, salsa basic step etc.). This deep learning approach willallow progressions of cluster elements across all the body’s landmarks to be analyzed, and the stochastic relationships it uncovered to construct adaptive and intuitive motion-to-music mappings across different genres to be applied. Mappings may then be calibrated to accommodate the user’s range of motion and may be periodically adjusted by the software or user. The resulting personalized mappings will allow users of all motion capabilities to naturally use their bodies to create expressive music that is intuitively related to their movements.

[0080] The music generation of the present invention can be done in either real-time for immediacy in the motion-powered CV-assisted music generation or off-line. The music generation of the present invention can be done pre or post the inclusion of motion from a user operating / playing the device of the present invention.

[0081] The device and system of the present invention can be used as a diagnostic or prediction tool wherein it monitors and analyzes relevant patient metrics to adjust / offer physiological “readings” or “measurements” for use in analysis of neuroplasticity or other neurobiological phenomena, physical interactions, emotional interactions, physiological interactions, social interactions, and / or can be combined with medical imaging data to derive useful correlations relevant to user outcomes.

[0082] The generative algorithm works, in particular with the “reconstruction” of a given song to take a digital symbolic construction of a song, such as a digital lead sheet (chords and melody), then it will dynamically & intelligently re-interpret that song in a new way. This can occur in a manner that is mediated by body motion, or completely independent of body motion (i.e. just hit the red button and a new version of the song plays).

[0083] For an example of this reconstruction, imagine giving a lead sheet of the song “Happy Birthday” to a professional jazz musician and asking them to play the song 10 different ways (in different styles, complexity levels, etc.). Each time, the song would sound substantially different, with different harmonic textures, rhythmic variations, and interpretations of the melody (including variations of the melody, and even substantial departures of that melody). The algorithm of the present invention is able to do this, with the computer inputs being the overall style, complexity, etc.

[0084] The present invention can also produce, without mediation from bodily movements, natural sounding, genre-specific music that is based on a minimalistic digital lead sheet (or chord / guitar chart, digital music chart) or symbolic description of the song, or text / voice prompts of musical characteristics. The algorithm is able to play novel versions of any song in any genre for therapeutic or non-therapeutic purposes. Thus, the song could or could not be influenced by computer-vision, and / or motion, and / or electronic / electrical brain signals.

[0085] The present invention may include a real-time visual representation of the user’s movements, the music being played, and the interactions of the two. The system of the present invention has a visualizer that aids the user’s understanding of their movement’s interaction with the musical parameters of the generative algorithm. The system can operate with or without this visualizer.

[0086] The present invention includes a provision that the technology can also allow users to specify certain aspects of its “motion parameters”-to-“music parameters” mapping so that their personal intuitions are realized. The present invention can predicate the motion-to-musicmapping, or the user could be given control to modify the parameters to suit their tastes or needs. This can be for therapeutic, aesthetic, or any other use the user needs / wants to use the device for.

[0087] The music generation logic is developed from stochastic relationships abstracted from analysis of audio and / or MIDI data from professional musicians playing songs in multiple genres and sub-genres at varying levels of harmonic, rhythmic, melodic, and temporal density and complexity, which can additionally be combined with analysis of written musical compositions for creating training data.

[0088] The present invention contains a visual display containing a real-time representation of a user’s body configured so that said user can see which parts of their body are being picked up / tracked, and a real-time visual feedback that shows said user how their motions and / or gestures are being interpreted and used to alter a resulting musical or sonified output. For example, when a user makes a closed fist gesture or a pointing finger gesture, the visual display may show an avatar of their hand changing to a particular color associated with each gesture, and text may appear indicating which musical parameter each gesture is associated with.

[0089] The CV will track not just the plurality of body parts, but also the relationships of these motions relative to one another. For example the way these motions are interacting will allow the present invention to derive a metric for kinetic energy that will influence the musical output.

[0090] According to the present invention, basic patent claims for adaptive mappings include but are not limited to: The ability of the system to periodically automatically recalibrate existing mappings personalized to the user based on a progressive or targeted outcome of a user; The ability of the system to automatically install new modular mappings; The ability of the system toallow users to design and implement new, customized mappings and upload supporting explanations and instructions.

[0091] For example, picture an athlete training to improve their skills or a patient rehabbing to regain lost skills. Initially, their skills will improve steadily, but after a while, as their bodies adapt to the training or rehabbing demands, their health progress will hit a plateau. The standard approach for pushing past this plateau is to increase the degree of challenge involved in the individual’s exercise regimen. And to do this, typically one (or more) of the following seven parameters is “ramped up”: frequency of the regimen, number of reps performed each time, speed of the performance, range of the motion carried out, weight of the resistance applied, amount of cognition required (e.g., number of steps required to solve a problem, or number of different concepts that must be synthesized for solving, etc.), and degree of accuracy required for a successful performance. It will be relatively easy for the present invention to progressively adjust these basic parameters when integrated into interventions (ie. therapeutic or rehabilitation games, puzzles, and exercises) in which a user performs predetermined movements or answers preset questions. For example, in a resistance exercise game in which a user stretches a resistance band, whenever the user's physical abilities adapt to a resistance level, the resistance band would be replaced by another resistance band with incrementally greater stiffness. In a quiz game in which a user answers questions that test their memory, whenever the user's cognitive abilities adjust to the difficulty level of the quiz, the quiz would be modified so that the number of mental steps needed to answer the new questions would be one more than before. In both cases, the instrument could adjust a basic parameter that corresponds to an incremental increase. However, it would be more difficult to use this type of approach to impose a system of“progressive adaptation” on less deterministic and more open-ended inventions into which the generative music and motion sonification engine is embedded. An example of this would be if the engine and related interventions are in “Musician Mode” or in “Freestyle Therapeutic Mode”. This is because it is not feasible to enumerate in advance the plethora of different configurations (directions, velocities, accelerations, etc.) in which a user could move their body parts while interacting with the engine and interventions such that rehabilitation or performance plateaus are either forestalled or quickly escaped.

[0092] According to the invention, the most practical way to progressively adjust the generative music and sonification engine so that it stays in-step with a user’s performance and / or health gains is to periodically alter the user parameters -to- musical parameters mappings that govern the ways in which the engine responds to the user’s interactions with it. For example, assume that the engine is in Therapeutic Mode, and a user with motor impairment is interacting with the engine through their random motions. Assume further that the health outcome “improved motor coordination” is a target, and so a “limb direction -to- instrument” mapping is in effect for the engine, in which the directions in which the user’s main limbs are moving are monitored, and the degree of synchronization in the directions will dictate the number of instruments that are layered into the generated sonic output, so that highly coordinated movements by the user generate rich orchestral sounds, while grossly uncoordinated movements produce just simple, rudimentary sounds by a single instrument.

[0093] Imagine that the user picks up on this relationship and tries to move their limbs in ways that enhance the musical or sonified experience they receive from the engine. As the user makes progress over time in synchronizing the motions of their limbs, the present invention’s systemwill automatically refine the engine’s mapping periodically so that a higher degree of motor coordination is needed to generate the same musical or sonified outcome. Progressively increasing the level of challenge that the engine presents the user is designed to sustain the health improvements that the user achieves for as long as possible. However, inevitably, the moment will arrive when the user’s motor coordination abilities will plateau. At this point, a new mapping that governs user-engine interactions will be needed to help the user break free from their plateau and resume progress.

[0094] To determine the identity this new mapping, the present invention will initiate two processes at that point. First, for Process #1, it will launch a trial-and-error operation in which the platform implements on a rotation basis a number of new user-engine interaction mappings to see whether one of them prompts renewed increases in the user’s motor coordination skills. The present invention includes this process as more users subscribe / adopt use of the platform and the reservoir of user / health data consequently increases. The platform will gain proficiency at this trial-and-error procedure and identify suitable new user-mapping very quickly. For example, with enough data, the platform will be able to match the user with a prior (anonymous) user who (i) is similar in many of the user characteristics that will have been discovered by then to be meaningful to the health or performance outcome “improved motor coordination”, and (ii) was observed to have benefitted from a particular user engine interaction mapping (e g. Mapping X, say) when they were at a similar point in thier rehabilitation arc as the current user is at now. Then, based on the match, the platform will quickly select Mapping X as the current user’s new interaction mapping. In other words, eventually, the present invention can / will efficiently and effectively select new mappings for plateaued users through ‘Expert Guided Modeling’. After anew mapping candidate is identified, a second process, Process #2, will initialize to personalize the candidate mapping to the user at hand. In this process, the user’s historical data will be analyzed to filter out any excessive correlation that any element of his / her past actions may have had with “improved motor coordination.”

[0095] For example, assume that Mapping X is a map in which certain dance moves that are made by a user (such as the Tango or the Waltz) cause the corresponding music (Tango music, Waltz music, etc.) to be played, and suppose that in Process #2, it is discovered that in the past, whenever the generative music engine included the violin as one of its instrumental sounds and in addition played a violin sequence that was hauntingly similar to a sequence in a ballet such as “Swan Lake”, the user would suddenly exhibit highly synchronized limb motions for the duration of the sequence. Suppose further that, upon additional analysis, it is noted that the aforementioned highly synchronized limb motions trace out portions of common ballet moves. Then, the platform would extract from these observations the intention to either automatically or with the help of the user or another supplementary / auxiliary assistance (e.g. care partner, practitioner, etc.) modify Mapping X so that the dance moves that are recognized and responded to by the modified map are ballet moves. Hence, as a result of Process #2, after the above intention is fulfilled, the user will be able to interact with a mapping that is not only new but highly personalized, and consequently, they will (very likely) be catapulted from their plateau into recommenced health or performance gains.

[0096] The present invention weaves the previously described capabilities into its generative music system.

[0097] In a first preferred embodiment there is a generative instrument for playing music from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voice-prompt, or a musical artifact used for creating a musical output comprising a computervision system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a realtime performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of said body of said user are being tracked and / or a real-time visual feedback that shows said user how at least one motion and / or gesture of said user are being interpreted and used to alter said musical output. Non-westernized representations of songs, like eastern microtonal compositions could be used as inputs as well.

[0098] In an alternative embodiment there is the addition of a knob / slider based control configured to operate said generative instrument. This knob / slider based control can be a traditional set of music controls or a computer generated or virtual version of a knob / slider based control.

[0099] In an alternative embodiment there is a generative instrument as in the first preferred embodiment that is further modified to capture a position, a velocity, an acceleration, and / or relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position. The present invention is configurable to track / monitor all body movements of a user and map those into a set of controls and modifications for the musical or sonified output.

[0100] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output. Body movement, and facial gestures are captured and mapped or converted to musical or sonified controls. The device of the present invention can also be used with electronic / electrical brain signals of a user / patient.

[0101] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical or sonified output. Any monitored or measurable condition of the user’s body can be used to alter the musical or sonified output by the system.

[0102] In an alternative embodiment there is a generative instrument as in the first preferred embodiment further configured to function either in real-time interactively or as a non-interactive automatic music generator (with all parameters specified at run-time) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said musical instrument and / or as specified by the user immediately prior to interacting with said musical instrument. The present invention can be used in any on or off line means to create a musical output.

[0103] In an alternative embodiment there is a generative instrument as in the first preferred embodiment further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity of a generated stream of music notes. Mapping of music controls and changes can include any modification of the music stream output by the system.

[0104] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said digital symbolic representation of a song or musical composition comprises of a harmonic structure and a melody of said song (or an equivalent non-western digital representation of a musical composition), and may be modified by text or voice-prompts of musical characteristics and / or a user-selected musical genre set or a non-western equivalent. The device of the present invention enables any body movement or condition to be used to control and create a musical output.

[0105] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said digital symbolic representation of a song or text or voice prompt mayonly provide minimalistic information about a harmony, a rhythm, a melody, an instrumentation set, a timbral characteristic, a musical characteristic, and / or a genre set, allowing said human motion to generate and mediate musical variations consistent with stochastic and rule-based representations of genre-specific musical and rhythmic patterns and user selected musical preferences. The device of the present invention can used with any music input content from a very small or short music or audio input to multiple complete songs or compositions and any other recorded and / or playable music content.

[0106] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of musical play or motion sonification. The progressively adaptive competence can be derived and included in any functional elements of the present invention.

[0107] In an alternative embodiment there is a generative instrument as in the first preferred embodiment further configured to map to a speech, a musical speech, a singing performance, and / or a sound effect in a deterministic manner, wherein said speech, said musical speech, said singing performance, and / or said sound effect are determined by an output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or a timbre transfer to give an expansive and dynamic range to said output that can be customized by said user. For example, this invention would enable the ethical use of a vocal clone of a famous singer to be used as an audio input, such that a user who desires theirmotion to be able to control the generative composition of a novel version of a song with a realistic audio output could generate such a composition that includes an audible representation of said famous singer singing lyrics to the song.

[0108] In an alternative embodiment there is a generative instrument as in the first preferred embodiment further configured to collect and record at least one data element based on at least one real-time interaction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user performance outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user performance outcome by analyzing any / all data accessible by said generative instrument, to suggest new or modified interactions with said generative instrument that the user should engage in to enable an improvement in said user performance outcome, and to provide an organized display of a user performance progress.

[0109] In an alternative embodiment there is a generative instrument as in the first preferred embodiment further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output. This sensor system can function in conjunction with, and / or separately from said CV-based motion sensing system, and can also be used to improve the motion detection itself viahigher resolution capture, or provide new types of information that are difficult or impossible to capture with CV alone.

[0110] In an alternative embodiment there is a generative instrument as in the first preferred embodiment wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV- mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

[0111] In a second preferred embodiment there is a generative instrument for playing neurorehabilitative music for engaging in digital music-based neurorehabilitative interventions from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voice-prompt, or a musical artifact creating a musical output comprising a computervision system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a musictransport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a realtime performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of said body of said user are being tracked and / or a real-time visual feedback that shows said user how a plurality of motions and / or gestures of said user are being interpreted and used to alter said musical output.

[0112] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to integrate with a digital neurorehabilitative intervention set designed to stimulate a neurochemical environment in a brain of said user that facilitates mechanisms of neuroplasticity, to analyze a health data of said user and / or a neurological health data of said user, to learn and remember changes in said health data and / or neurological health data of said user and an engagement of said user with said neurorehabilitative music that reflect changes in a neurological health outcome of said user, and to progressively adapt said musical output and said engagement of said user with said neurorehabilitative music and a music-based neurorehabilitative intervention set with machine learning to facilitate or target a therapeutic progress of said user using any combination of a physical interaction, an emotional interaction, a cognitive interaction, a sensory interaction, a social interaction, and / or a physiological interaction.

[0113] In an alternative embodiment there is a generative instrument as in the second preferred embodiment wherein said physical interaction, said emotional interaction, said cognitive interaction, said sensory interaction, said social interaction, and / or said physiological interaction are combined with a medical image data to develop statistical correlations of a health and / or neurological health impact of music. Any and all body movements and conditions can be used to control the musical output.

[0114] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further comprising a knob / slider based control configured to operate said generative instrument.

[0115] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to capture a position, a velocity, an acceleration, and / or relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position.

[0116] In an alternative embodiment there is a generative instrument as in the second preferred embodiment wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output.

[0117] In an alternative embodiment there is a generative instrument as in the second preferred embodiment wherein said plurality of key tracking points further comprises a plurality of metricsfor said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

[0118] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to function in either real-time interactively or as a non-interactive automatic music generator (with all parameters specified at run-time) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said musical instrument and / or as specified by the user prior to interacting with said musical instrument.

[0119] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity of a generated stream of music notes.

[0120] In an alternative embodiment there is a generative instrument as in the second preferred embodiment wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy.

[0121] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to map to speech, musical speech, singing, and / or sound effects in a deterministic manner, wherein said speech, musical speech, singing, and / or sound effects aredetermined by the output specified by the mapping system and personalized to said user, and may use natural language processing algorithms and timbre transfers to give a dynamic range of audible outputs that may be customized by said user. For example, say a child with non-verbal autism wants to use the platform to use gestures to “sing” a song with sign language to use music to communicate audibly and as a neurorehabilitation tool to help improve their brain’s capacity for speech. The child would, within a CV environment, perform gestures related to known sign languages such as American Sign Language or gestures of his / her own that he / she personally associates with any object, action, event, and so on. For instance, the child might gesture the phrase “Mary lamb little”. A natural language processor would, in real-time, convert the gestures to speech, vocalizing the sentence “Mary had a little lamb” in a voice that has timbral characteristics that the child likes (i.e. a selected voice clone audio input with timbral transfer modifications, or a voice clone of the child’s favorite music artist used with permission from said artist), while the music engine would play, in the background, a version of the song Mary Had a Little Lamb, with the music generation guided by a lead sheet of the song stored in the bank of cataloged musical compositions.

[0122] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to collect and record at least one data element based on at least one real-time interaction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user neurorehabilitation outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user neurorehabilitation outcome by analyzing any / all data accessible by said generative instrument, to enable improvement suggestions of new or modifiedinteractions with said generative instrument that the user should engage in to enable an improvement in said user neurorehabilitation outcome, and to provide an organized display of a user neurorehabilitation progress.

[0123] In an alternative embodiment there is a generative instrument as in the second preferred embodiment further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output. This sensor system can function in conjunction with, and / or separately from said CV-based motion sensing system, and can also be used to improve the motion detection itself via higher resolution capture, or provide new types of information that are difficult or impossible to capture with CV alone.

[0124] In an alternative embodiment there is a generative instrument as in the second preferred embodiment wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

[0125] In a third preferred embodiment there is a generative bioinformatics treatment / therapy instrument for playing music from a digital symbolic representation of a musical composition or song, an audio input or fde, a text or voice-prompt, and / or a musical artifact creating a musical output comprising a computer-vision system configured to capture a human motion, a data- driven digital bioinformatics system for evaluating a user / patient health and targeting a user / patient health outcome, a mapping system having a data acquisition system with a plurality of data elements, one of said plurality of data elements being said user / patient outcome configured to translate said human motion into a plurality of musical parameters adjusted by said user / patient outcome, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a real-time performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user / patient computer-vision control, and a visual display containing a real-time representation of a body of a user / patient configured so said user / patientcan see which parts of said body of said user / patient are being tracked and / or a real-time visual feedback that shows said user / patient how a plurality of motions and / or gestures of said user / patient are being interpreted and used to alter said musical output wherein said mapping system is configured to adjust said plurality of musical parameters based on said plurality of data elements.

[0126] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further comprising a knob / slider based control configured to operate said generative instrument.

[0127] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to collect and record at least one data element altered by at least one real-time user / patient event, to identify and remember at least one musical characteristic with a variable strength association to a positive user / patient health outcome, to progressively adapt said generative treatment / therapeutic instrument to identify a change in said health outcome of said user / patient by analyzing any / all relevant health and rehabilitation data accessible by said generative treatment / therapy instrument, to facilitate targeted improvements in said user / patient health, and to provide an organized display of user / patient health / therapeutic / rehabilitative progress.

[0128] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to capture a position, a velocity, an acceleration, and relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user / patient, one of said plurality of key tracking points being a head position and / or a hand position.

[0129] In an alternative embodiment there is a generative instrument as in the third preferred embodiment wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user / patient which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user / patient associated with mood and emotion to influence and / or alter said musical output.

[0130] In an alternative embodiment there is a generative instrument as in the third preferred embodiment wherein said plurality of key tracking points further comprises a plurality of metrics for said user / patient, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

[0131] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to either function in a real-time interactive mode or in a non- interactive automatic music generator mode (with all parameters specified at run-time) to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation in a pre-mapping process or a post-mapping process prior to creating said musical output.

[0132] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transportcontrols, an effects depth, a set of dynamic parameters, a synthesizer parameter, and / or a smart parameter that control a complexity of a generated stream of music notes.

[0133] In an alternative embodiment there is a generative instrument as in the third preferred embodiment wherein said generative treatment / therapy instrument is configured to be progressively adaptive, changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of music played or data sonified.

[0134] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to map to a speech, a musical speech, a singing performance, and / or a sound effect in a deterministic manner, wherein said speech, said musical speech, said singing performance, and / or said sound effect are determined by an output specified by the mapping system and personalized to said user / patient, and to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or and a timbre transfer to give an expansive and dynamic range to said output that can be customized by said user / patient. For example, perhaps a user wishes to be motivated to continue therapy with encouragement from vocalized personalized musical speech that is tied to their specific training and progress.

[0135] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to collect and record at least one data element based on at least one real-time interaction with said generative treatment / therapy instrument by said user / patient, to allow an identification of a variable strength association with a positive user / patient health outcome and store said data element with said identification, to progressively adapt saidgenerative treatment / therapy instrument based on an identified change in a user / patient health outcome by analyzing any / all data accessible by said generative treatment / therapy instrument, to enable improvement suggestions in said user / patient health outcome, and to provide an organized display of a user / patient health progress.

[0136] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output. This sensor system can function in conjunction with, and / or separately from said CV-based motion sensing system, and can also be used to improve the motion detection itself via higher resolution capture, or provide new types of information that are difficult or impossible to capture with CV alone.

[0137] In an alternative embodiment there is a generative instrument as in the third preferred embodiment wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, healthoutcomes, and / or other types of characteristics or outcomes from which the generative CV- mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

[0138] In an alternative embodiment there is a generative instrument as in the third preferred embodiment further configured to collect and record said plurality of data elements based on a plurality of real-time user movements, and map said data elements to sound effects in a deterministic manner, wherein said sound effect are determined by an output specified by the mapping system and personalized to said user, to give said user an intuitive real-time auditory feedback about said user movements so that said user can use said feedback to modify in realtime said user movements.

[0139] For example, consider a recreational tennis player who has tennis elbow but would like to continue playing. To avoid worsening their injury, and in fact, to support its healing, it is recommended that to transfer momentum to the ball while serving, the player should not try to generate a large force with their arm, but instead should attempt to generate a coordinated sum of many small forces along their body’s entire kinetic chain. In other words, the player should try to follow a recommended whole-body service technique.

[0140] Under the configuration of the generative instrument that is being described in this section, the player would be able to practice the recommended technique by interacting with theinstrument as follows: (i) First, arrange that the instrument’s movement-to-sound effect mapping be personalized so that whole-body movements that are “far” from the recommended service motion generate loud sound effects, while whole-body movements that are “near” to the recommended service motion generate commensurately soft sound effects, (ii) Second, practice service motions in the instrument’s CV environment and use the auditory feedback to adjust the service motions in real-time to minimize the overall volume of the sound effects generated. The player is expected to acquire the recommended service technique with only a relatively small number of repetitions of this process and quickly return to the courts to play the game in a safe manner.

[0141] The hallmark the present invention is a proprietary, data-driven “smart instrument” that uses computer-vision to map physical movements onto enjoyable musical compositions or sonify motion, in the context of neurorehabilitative interventions that can be progressively customized to the user’s sensitive capabilities, therapeutic needs, and musical tastes. This smart instrument of the present invention provides many benefits to users that other therapeutic platforms and music-based rehabilitation methods do not. Benefits of the present invention include:

[0142] The smart instrument of the present invention leverages computer-vision (CV), which may function with only a traditional webcam on a personal computing device. This is advantageous as webcams are highly ubiquitous and come pre-installed in most laptops & smartphones nowadays. Moreover, it eliminates the use of external physical motion sensor devices, which are cumbersome and provide the same information as CV. CV is also advantageous as it offers the potential for a more expansive and dynamic range of inputcontrollers, when compared to hardware controllers (e.g., physical knobs / sliders), because the CV can also be updated continuously through software and will not degrade over time.

[0143] The generative music engine of the present invention provides infinite variability to the musical output. This generative engine is also particularly advantageous in comparison to prerecorded music as it allows a musical or sonified output that is highly flexible and can adapt in incredibly precise and specific ways to the user’s input.

[0144] The platform is advantageous to traditional therapeutic (both music-based and nonmusical) approaches in that it harnesses the known benefits of music in memory recall. Consider the example of a patient that can still recognize / play music but has lost many other functions. The active creation of music by users with their bodily movements will maximize distributed cross-hemi spheral neural activity in the user and stimulate a neurochemical environment that increases their brain’s capacity for neuroplasticity and mechanisms of neurological repair.

[0145] The versions of nostalgic, error-free, professional-sounding musical compositions that users will be able to generate by interacting with this smart instrument, with control over genre and song and a multitude of musical characteristics, will give them a personalized, aesthetically pleasing experience that is expected to elevate their mood and self-esteem for sustained periods.

[0146] A curriculum of neurorehabilitative interventions developed with leading Neuroscientists & Neurologic Music Therapists will be used as the basis for specifying certain modes of interactions with the smart instrument that will be prescribed to users, and the interactions will target specific therapeutic objectives, such as improving their memory, motor skills, and proprioception.

[0147] Software that records data on real-time patient events, identifies musical characteristics with strong associations with positive patient outcomes, and provides organized displays of patient progress will be integrated with our smart instrument, and the precise, tailored therapeutic recommendations regarding a user’s rehabilitation regimen that will flow from this integration will maximize the efficiency and effectiveness of a user’s music-based therapy or treatment.

[0148] The smart instrument of the present invention will also be connected to an algorithm that utilizes a user’s data to progressively adjust the instrument’s mode of interacting with him / her to offer increasingly personalized and continuously challenging therapy, and as a result, the user is expected to maintain a strong interest in interacting with the smart instrument and adhering to a rehabilitation regimen that integrates its use.

[0149] The smart instrument of the present invention will allow a person to access a dynamic range of vocalized musical expression by using gestures with customizable and personalized audio processing (timbre transfers and natural speech processing) for musical speech or singing.

[0150] The invention has been described by way of examples only. Therefore, the foregoing is considered as illustrative only of the principles of the invention. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact construction and operation shown and described, and accordingly, all suitable modifications and equivalents may be resorted to, falling within the scope of the claims.

[0151] Although the invention has been explained in relation to various embodiments, it is to be understood that many other possible modifications and variations can be made without departing from the spirit and scope of the invention.

Claims

CLAIMS1. A generative instrument for playing music from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voice-prompt, or a musical artifact used for creating a musical output comprising a computer-vision system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a realtime performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of said body of said user are being tracked and / or a real-time visual feedback that shows said user how at least one motion and / or gesture of said user are being interpreted and used to alter said musical output.

2. The generative instrument of claim 1 further comprising a knob / slider-based control configured to operate said generative instrument.

3. The generative instrument of claim 1 further configured to capture a position, a velocity, an acceleration, and / or a relationship of limb and facial feature movement in a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position.

4. The generative instrument of claim 3 wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output.

5. The generative instrument of claim 3 wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

6. The generative instrument of claim 1 further configured to function either in real-time interactively or as a non-interactive automatic music generator with parameters specified at and / or prior to run-time to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as definedduring the design of said generative instrument and / or as specified by the user prior to interacting with said generative instrument.

7. The generative instrument of claim 1 further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity or a configuration of a generated stream of music notes.

8. The generative instrument of claim 1 wherein said digital symbolic representation of a song or musical composition comprises of a harmonic structure and a melody of said song, and may be modified by text or voice-prompts of musical characteristics and / or a user-selected musical genre set or a non-western equivalent.

9. The generative instrument of claim 1 wherein said digital symbolic representation of a song, a text, or a voice prompt may only provide minimalistic information about a harmony, a rhythm, a melody, an instrumentation set, a timbral characteristic, a musical characteristic, and / or a genre set, allowing said human motion to generate and mediate musical variations consistent with stochastic and rule-based representations of genre-specific musical and rhythmic patterns and user selected musical preferences.

10. The generative instrument of claim 1 wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of musical play.

11. The generative instrument of claim 1 further configured to map to a speech, a musical speech, a singing performance, and / or a sound effect in a deterministic manner, wherein said speech, said musical speech, said singing performance, and / or said sound effect are determined by an output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or a timbre transfer to give an expansive and dynamic range to said output that can be customized by said user.

12. The generative instrument of claim 1 further configured to collect and record at least one data element based on at least one real-time interaction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user performance outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user performance outcome by analyzing any / all data accessible by said generative instrument, to suggest new or modified interactions with said generative instrument that the user should engage in to enable an improvement in said user performance outcome, and to provide an organized display of a user performance progress.

13. The generative instrument of claim 1 further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement a collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output.

14. The generative instrument of claim 1 wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of- use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

15. A generative instrument for playing neurorehabilitative music for engaging in digital musicbased neurorehabilitative interventions from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voice-prompt, or a musical artifact creating a musical output comprising a computer- vision system configured to capture a plurality of human motions, one of said plurality of human motions being a hand motion, a mapping system configured to translate said human motion into a plurality of musical parameters, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a realtime performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user computer-vision control, and a visual display containing a real-time representation of a body of a user configured so said user can see which parts of said body of said user are being tracked and / or a real-time visual feedback that shows said user how a plurality of motions and / or gestures of said user are being interpreted and used to alter said musical output.

16. The generative instrument for playing neurorehabilitative music of claim 15 further configured to integrate with a digital neurorehabilitative intervention set designed to stimulate a neurochemical environment in a brain of said user that facilitates mechanisms of neuroplasticity,to analyze a health data of said user and / or a neurological health data of said user, to learn and remember changes in said health data and / or neurological health data of said user and an engagement of said user with said neurorehabilitative music that reflect changes in a neurological health outcome of said user, and to progressively adapt said musical output and said engagement of said user with said neurorehabilitative music and a music-based neurorehabilitative intervention set with machine learning to facilitate or target a therapeutic progress of said user using any combination of a physical interaction, an emotional interaction, a cognitive interaction, a sensory interaction, a social interaction, and / or a physiological interaction.

17. The generative instrument for playing neurorehabilitative music of claim 16 wherein said physical interaction, said emotional interaction, said cognitive interaction, said sensory interaction, said social interaction, and / or said physiological interaction are combined with a medical image data to develop statistical correlations of a health and / or neurological health impact of music.

18. The generative instrument for playing neurorehabilitative music of claim 15 further comprising a knob / slider based control configured to operate said generative instrument.

19. The generative instrument for playing neurorehabilitative music of claim 15 further configured to capture a position, a velocity, an acceleration, and / or relationships of limb andfacial feature movement of a plurality of key tracking points on said body of said user, one of said plurality of key tracking points being a head position or a hand position.

20. The generative instrument for playing neurorehabilitative music of claim 19 wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user associated with mood and emotion to influence and / or alter said musical output.

21. The generative instrument for playing neurorehabilitative music of claim 19 wherein said plurality of key tracking points further comprises a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

22. The generative instrument for playing neurorehabilitative music of claim 15 further configured to function in either real-time interactively or as a non-interactive automatic music generator to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said musical instrument and / or as specified by the user prior to interacting with said musical instrument.

23. The generative instrument for playing neurorehabilitative music of claim 15 further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamics, a synthesizer parameter, and / or a smart parameter to control a complexity or a configuration of a generated stream of music notes.

24. The generative instrument for playing neurorehabilitative music of claim 15 wherein said generative instrument is configured to be progressively adaptive changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy.

25. The generative instrument for playing neurorehabilitative music of claim 15 further configured to map to a speech, a musical speech, a singing performance, and / or a sound effect in a deterministic manner, wherein said speech, said musical speech, said singing performance, and / or said sound effect are determined by an output specified by the mapping system and personalized to said user, and are further configured to use a natural language processing algorithm, an effects depth, a set of dynamics, a synthesizer parameter, a smart parameter, and / or a timbre transfer to give an expansive and dynamic range to said output that can be customized by said user.

26. The generative instrument for playing neurorehabilitative music of claim 15 further configured to collect and record at least one data element based on at least one real-timeinteraction with said generative instrument by said user, to allow an identification of a variable strength association with a positive user neurorehabilitation outcome and store said data element with said identification, to progressively adapt said generative instrument based on an identified change in a user neurorehabilitation outcome by analyzing any / all data accessible by said generative instrument, to enable improvement suggestions of new or modified interactions with said generative instrument that the user should engage in to enable an improvement in said user neurorehabilitation outcome, and to provide an organized display of a user neurorehabilitation progress.

27. The generative instrument for playing neurorehabilitative music of claim 15 further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computer-vision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output.

28. The generative instrument for playing neurorehabilitative music of claim 15 wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts ormenu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

29. A generative bioinformatics treatment / therapy instrument for playing music from a digital symbolic representation of a musical composition or song, an audio input or file, a text or voiceprompt, or a musical artifact creating a musical output comprising a computer-vision system configured to capture a human motion, a digital bioinformatics system for evaluating a user / patient health and targeting a user / patient health outcome, a mapping system having a data acquisition system with a plurality of data elements, one of said plurality of data elements being said user / patient outcome configured to translate said human motion into a plurality of musical parameters adjusted by said user / patient outcome, one of said plurality of musical parameters being a music transport control, a processor system having a generative engine configured to intake a symbolic representation of music or songs such as streams of chord changes and to create / enable a real-time performance of a realistic harmony / chord voicing, a voice leading, an accompaniment generation, a rhythmic generation, a melody generation, a genre set selection, a timbre generation, a natural sounding speech, a musical speech, a singing performance, and / or a musical instrumentation set selection which is connected to a real-time user / patient computer-vision control, and a visual display containing a real-time representation of a body of a user / patient configured so said user / patient can see which parts of said body of said user / patient are being tracked and / or a real-time visual feedback that shows said user / patient how a plurality of motions and / or gestures of said user / patient are being interpreted and used to alter said musical output wherein said mapping system is configured to adjust said plurality of musical parameters based on said plurality of data elements.

30. The generative treatment / therapy instrument of claim 29 further comprising a knob / slider based control configured to operate said generative treatment / therapy instrument.

31. The generative treatment / therapy instrument of claim 29 further configured to collect and record at least one data element altered by at least one real-time user / patient event, to identify and remember at least one musical characteristic with a variable strength association to a positive user / patient health outcome, to progressively adapt said generative treatment / therapeutic instrument to identify a change in said health outcome of said user / patient by analyzing any / all relevant health and rehabilitation data accessible by said generative treatment / therapyinstrument, to facilitate targeted improvements in said user / patient health, and to provide an organized display of user / patient health / therapeutic / rehabilitative progress.

32. The generative treatment / therapy instrument of claim 29 further configured to capture a position, a velocity, an acceleration, and relationships of limb and facial feature movement of a plurality of key tracking points on said body of said user / patient, one of said plurality of key tracking points being a head position and / or a hand position.

33. The generative treatment / therapy instrument of claim 32 wherein said plurality of key tracking points further comprises any combination of a plurality of body parts of said user / patient which provide expressions of mood including at least a facial feature and / or a body posture, and said mapping system is further configured to enable a physical body language of said user / patient associated with mood and emotion to influence and / or alter said musical output.

34. The generative treatment / therapy instrument of claim 32 wherein said plurality of key tracking points further comprises a plurality of metrics for said user / patient, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said mapping system is further configured to enable said plurality of metrics to influence and / or alter said musical output.

35. The generative treatment / therapy instrument of claim 29 further configured to either function in a real-time interactive mode or in a non-interactive automatic music generator mode with allparameters specified at run-time to enable a performance of a motion-powered computer-vision assisted music generation, wherein said mapping system is further configured for operation as defined during the design of said generative treatment / therapy instrument and / or as specified by said user / patient prior to interacting with said generative treatment / therapy instrument.

36. The generative treatment / therapy instrument of claim 29 further configured to map to musical parameters that control a real-time generation and a texture of a musical performance itself, including any combination of said music transport controls, an effects depth, a set of dynamic parameters, a synthesizer parameter, and / or a smart parameter that control a complexity or a configuration of a generated stream of music notes.

37. The generative treatment / therapy instrument of claim 29 wherein said generative treatment / therapy instrument is configured to be progressively adaptive, changing any combination of a frequency, a number of reps, a speed, a range of movement, a resistive load, a cognitive taxation, a number of moves, and / or an accuracy of music played.

38. The generative treatment / therapy instrument of claim 29 further configured to map to a speech, a musical speech, a singing performance, and / or a sound effect in a deterministic manner, wherein said speech, said musical speech, said singing performance, and / or said sound effect are determined by an output specified by the mapping system and personalized to said user / patient, and to use a natural language processing algorithm, an effects depth, a set of dynamics, asynthesizer parameter, a smart parameter, and / or and a timbre transfer to give an expansive and dynamic range to said output that can be customized by said user / patient.

39. The generative treatment / therapy instrument of claim 29 further configured to collect and record at least one data element based on at least one real-time interaction with said generative treatment / therapy instrument by said user / patient, to allow an identification of a variable strength association with a positive user / patient health outcome and store said data element with said identification, to progressively adapt said generative treatment / therapy instrument based on an identified change in a user / patient health outcome by analyzing any / all data accessible by said generative treatment / therapy instrument, to enable improvement suggestions in said user / patient health outcome, and to provide an organized display of a user / patient health progress.

40. The generative treatment / therapy instrument of claim 29 further configured to integrate with a wearable sensor set and a mapping system designed to augment or supplement a computervision assisted detection of user motion and / or a relationship of user limb, hand, facial feature, and / or bodily movement, to augment or supplement the collection of a plurality of metrics for said user, said plurality of metrics including a heartbeat, an oxygen level, an electronic / electrical brain signal, and an autonomic nervous system marker wherein said wearable sensor set and mapping system are further configured to enable said user motion and plurality of metrics to influence and / or alter said musical output.

41. The generative treatment / therapy instrument of claim 29 wherein said digital symbolic representations of a musical composition or song, an audio input or file, a text or voice-prompt, and / or a musical artifact, are stored and catalogued by titles, musical characteristics, and / or other characteristics that the user may access via voice or text prompts or menu selection options to facilitate ease-of-use and improved access to databases of musical compositions associated with musical characteristics, affect characteristics, health outcomes, and / or other types of characteristics or outcomes from which the generative CV-mediated musical compositions could be developed and / or influenced in manners that are relevant for their target use case and interactions with the system and to allow the system to leverage machine learning and Al to assist the users selection of characteristics that may modify and / or improve the user interactions and mapping adaptations for the intended use case, such that the user experience, user musical interactions, performance objectives, and creative engagement are more easily personalized.

Images