Automatic evaluation of haptic-based rhythmic performance with reference to musical scores

Abstract

A technique is described for automated evaluation of haptic-based rhythmic performance with reference to a musical score. An audiovisual representation of real-time progression through the musical score is output to a user. The output corresponds to the progression of a sequence of scored rhythmic (SNR) events at a tempo. During the output, haptic performance (HP) events are received from the user via a haptic interface as the user's rhythmic performance of the SNR events at the tempo. The automated technique evaluates the user's performance to determine a rhythmic match between the received HP events and evaluation data of the SNR events. Visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION

[0001] Embodiments relate generally to performance feedback applications, and more particularly to the automated evaluation of rhythmic haptic performance events relative to notated musical score events. [Background technology]

[0002] The ability to sight-read musical rhythms is a crucial skill in music education at all levels. Standard music notation includes a large symbol system for specifying rhythmic features. For example, a typical music score depicts musical events spanning linear time. A musical event can include at least a note (generally a designation of filled musical space, such as by pitch) and a rest (generally a designation of empty musical space, such as by silence). Each musical event can be notated on a music score in different forms to indicate the duration of the musical event. For example, a particular note or rest can be designated as consuming a duration such as a whole note, half note, quarter note, eighth note, dotted quarter note, double dotted quarter note, triplet, etc. By mapping the notated duration to a particular tempo (which may also be specified by the music score, by the music genre, or in some other way), a musician can know how long to hold the note or rest.

[0003] Learning to read and play such rhythmic patterns generally can follow multiple stages. In one stage, a musician learns the notations and their explicit meanings. For example, a beginner-level musician may learn about quarter notes, the relationship between their duration and the duration of a measure based on one or more time signatures, the relative duration of the quarter note to other elements of the measure or song, etc. In another stage, a musician may learn to play quarter notes, such as by practicing the feel of playing quarter notes at one or more tempos. In another stage, a musician may learn and practice rhythmic patterns that include quarter notes and other musical elements (e.g., quarter rests, half notes, etc.). In another stage, a musician may learn to play rhythmic patterns in the context of a musical composition that includes additional musical features such as pitch, dynamics, etc. Still, more advanced players may encounter difficult and / or unfamiliar rhythmic patterns and may revisit one or more of the same stages.

[0004] Many conventional techniques have been used by music students for centuries to learn rhythmic elements and patterns. More recently, several computer-based music training applications have incorporated similar techniques that allow students to learn music without a live teacher. Some such applications train users to play musical pieces where rhythm is learned only in the context of a particular piece. For example, a recording of a piece is played for the user as a small portion of the musical score (e.g., one or two measures of the musical score), and the user attempts to copy what they hear, including the pitch and rhythm. While such applications can help students learn a particular piece, students may struggle to transfer or extend that knowledge to new pieces and / or new rhythms, and they may find that each new piece is a new exercise in imitation. Furthermore, users of such applications may learn to play numerous pieces by only encountering relatively small variations in rhythm. Other such applications teach rhythm decontextualized from any musical piece. For example, a recorded rhythmic pattern may play by itself, synchronized with flashing graphics, etc., and the user may be prompted to repeat the rhythmic pattern via the interface. While such applications can help students improve their rhythmic recall and can efficiently provide users with a wide variety of rhythmic patterns, students using such applications may never know how such rhythms are notated in the context of a musical piece. These and other conventional music training applications tend to be limited in their ability to provide students (especially more experienced students) with effective rhythmic training contextualized to standard sheet music notation of musical pieces. Summary of the Invention [Means for solving the problem]

[0005]

[0005] An embodiment of the present invention relates to automated haptic-based rhythmic performance evaluation with reference to a musical score. An audiovisual representation of real-time progression through a musical score is output to a user. The output corresponds to the progression of a sequence of scored rhythmic (SNR) events at a tempo. During the output, haptic performance (HP) events are received from the user via a haptic interface as the user's rhythmic performance of the SNR events at that tempo. An automated technique evaluates the user's performance to determine a rhythmic match between the received HP events and evaluation data of the SNR events. Visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

[0006] According to one set of embodiments, a method is provided for haptic-based assessment of rhythmic performance with reference to a musical score, the method comprising the steps of: outputting, via an output device interface of a computing device, an audiovisual representation of real-time progression through the musical score at a tempo, the musical score including a musical notation representation of each of a sequence of score-notated rhythmic (SNR) events, each SNR event having a stored logical definition including an associated SNR onset time and an associated SNR offset time; acquiring, during the outputting step, a plurality of haptic performance (HP) events corresponding to user interactions with the haptic device interface of the computing device representing the user's rhythmic performance of the musical score at the tempo, each HP event having an associated HP onset time and an associated HP offset time; and acquiring, for each SNR event of the plurality of SNR events, an associated assessment metric of the SNR event based on the associated SNR onset time and the associated SNR offset time. generating evaluation data for the plurality of SNR events by a rhythm evaluator subsystem in communication with the output device interface and the haptic device interface by comparing the HP onset time and associated evaluation criteria for each SNR event with an associated HP onset time and an associated HP offset time of each of at least some of the plurality of HP events, the comparing including: identifying the plurality of matching events as those of the HP events each determined to correspond in time to one of the plurality of SNR events; and evaluating a rhythmic compatibility between each of the plurality of matching events and a temporally corresponding one of the plurality of SNR events; and displaying visual feedback via the output device interface for graphically mapping the evaluation data of the plurality of SNR events to a music notation representation of the plurality of SNR events on the musical score.

[0007]

[0007] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter, which subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0008]

[0008] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0009]

[0009] The present disclosure is described in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an exemplary haptic rhythmic training system according to various embodiments described herein. [Figure 2] FIG. 1 illustrates another exemplary haptic rhythmic training system implemented in a cloud-based environment, according to various embodiments described herein. [Figure 3A] 10A-10C illustrate example screenshots from an example application for implementing features of a haptic rhythmic training system according to various embodiments described herein. [Figure 3B] FIG. 10 is another diagram illustrating an example screenshot from an example application for implementing features of a haptic rhythm training system according to various embodiments described herein. [Figure 3C] FIG. 10 is yet another diagram illustrating an example screenshot from an example application for implementing features of a haptic rhythm training system according to various embodiments described herein. [Figure 4] FIG. 1 is a schematic diagram of one embodiment of a computer system capable of implementing various system components and / or performing various steps of methods provided by various embodiments. [Figure 5] FIG. 1 is a flow diagram of an exemplary method for haptic-based assessment of a rhythmic performance with reference to a musical score, according to various embodiments. [Figure 6A] 10A-10C illustrate example screenshots from an example application for implementing historical user feedback functionality of a haptic rhythm training system, according to various embodiments described herein. [Figure 6B] FIG. 10 is another diagram illustrating an example screenshot from an example application for implementing historical user feedback functionality of a haptic rhythm training system according to various embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0011]

[0016] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a second label (e.g., lowercase) that distinguishes between the similar components. When only a first reference label is used herein, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label.

[0012]

[0017]

[0023] Embodiments of the disclosed technology will become clearer when considered in conjunction with the following description of the drawings. In the following description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, those skilled in the art should recognize that the present invention may be practiced without these specific details. In some instances, circuits, structures, and techniques are not shown in detail to avoid obscuring the present invention.

[0013]

[0018] The ability to sight-read musical rhythms is a crucial skill in music education at all levels. A typical music score includes a notation of musical events spanning linear time. Music sight-reading includes the real-time performance of these musical events. For example, a typical music score may include one or more staves on which are arranged at least notes (typically the designation of filled musical space, such as by pitch) and rests (typically the designation of empty musical space, such as by silence). For most instruments, the vertical dimension of the staff relates to pitch, and the horizontal dimension of the staff relates to time. A music score may additionally include designations of time signatures, key signatures, dynamics, and / or many other musical features.

[0014]

[0019] As used herein, notes and rests are referred to as score-notated rhythmic (SNR) events, as they are notated in a musical score. Each SNR event can include an associated SNR onset time and an SNR offset time. As used herein, "onset time" refers to the time a rhythmic event begins, and "offset time" refers to the time a rhythmic event ends. The relative horizontal placement of the SNR event on the staff can indicate the SNR onset time, and a standard notation symbol specifies the duration of the SNR event, thereby indicating the SNR offset time. Standard music notation includes a large symbology for specifying SNR events. For example, various symbols are used to indicate that a note or rest occupies a duration such as a whole note, half note, quarter note, eighth note, dotted quarter note, double dotted quarter note, eighth note triplet, quarter note tied to a sixteenth note, etc. The played duration of an SNR event can be determined by a combination of the notated duration (e.g., quarter note) and the tempo (e.g., beats per minute). In some cases, performance duration is determined by additional and / or alternative factors such as musical genre, time signature, and additional score markings (e.g., indicating to play a particular passage more quickly, to gradually slow down a passage, etc.) By recognizing and understanding all of the rhythmic symbology, a musician can know when to play each SNR event and how long to hold each SNR event, thereby transforming notated music into performed music.

[0015]

[0020] The ability to convert notated SNR events into performed SNR events, especially in real time, can involve extensive training and practice. Even skilled musicians can struggle to perform complex and / or unfamiliar rhythmic patterns. Embodiments herein provide novel techniques for providing rhythmic training to musicians by providing real-time, score-referenced associations between SNR events provided by an application and haptic performance (HP) events provided by a user. An audiovisual representation of real-time progression through a musical score (including progression through a sequence of SNR events at a tempo) is output to the user. In some implementations, the tempo is user-controllable. For example, the user can speed up or slow down the tempo using a slider bar or any other suitable interface control. Meanwhile, HP events are received from the user corresponding to user interaction with a haptic device interface representing the user's rhythmic performance of the SNR events at the tempo. The user's performance is evaluated to determine a rhythmic match between the received HP events and evaluation data for the SNR events, and visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

[0016]

[0021] Referring initially to FIG. 1 , an exemplary haptic rhythm training system 100 is shown in accordance with various embodiments described herein. System 100 includes a device input / output (I / O) subsystem 105, a rhythm evaluator subsystem 150, and a music score (MS) data store 140. An embodiment of device I / O subsystem 105 includes a haptic interface 110 having a haptic event processor 115, a display interface 120 having a display processor 125, and an audio interface 130 having an audio processor 135. An embodiment of system 100 is implemented on a computing device. In some embodiments, the computing device is a portable electronic device such as a laptop computer, a tablet computer, or a smartphone. In other embodiments, the computing device is an appliance such as a desktop computer. In other embodiments, the computing device is integrated with a musical instrument such as an electronic piano.

[0017]

[0022] Embodiments of the haptic rhythmic training system 100 provide a rhythmic training environment based on detecting and evaluating rhythmic correspondences between a user's rhythmic performance (based on detected haptic performance (HP) events) and an audio-visual representation of real-time progression through a sequence of score-notated rhythmic (SNR) events notated on a musical score. Details of the musical score, including the sequence of SNR events, are stored by the MS data store 140. As shown, the MS data store 140 stores at least MS visual data 145 and MS logical data 143. In some embodiments, the MS visual data 145 includes all information used by the display processor 125 to generate a graphical score representation, including the graphical representation of the SNR events, for visual output by the display interface 120. For example, the MS visual data 145 includes a stored graphical representation of the score, including notes and rests, and any additional graphical score elements (e.g., staff lines, bar lines, note symbols, accidentals, time signatures, key signatures, dynamics, tempo symbols, expression symbols, titles, lyrics, etc.). In other embodiments, MS visual data 145 includes only a portion of the information used by display processor 125 to generate a visual output of the musical score via display interface 120. For example, MS visual data 145 includes score elements other than notes and rests, and display processor 125 generates a visual representation of the notes and rests based on MS logical data 143 for SNR events.

[0018]

[0023] MS logical data 143 logically defines at least a sequence of SNR events, indicating at least an associated SNR onset time and SNR offset time for each SNR event. In some implementations, MS logical data 143 for each SNR event includes a type identifier indicating whether the SNR event represents a filled musical space (e.g., a note) or an empty musical space (e.g., a rest). In some implementations, MS logical data 143 includes other information about a particular SNR event, such as the pitch of the note, whether the SNR event is part of a triplet or other grouping, whether the SNR event is associated with a particular hand or finger, etc.

[0019]

[0024] In some embodiments, MS logical data 143 is score-referenced, such that the SNR onset time and / or SNR offset time (and / or SNR duration) are defined with respect to the notation of the underlying musical score. The relationship to the score can be based on the type of note or rest (e.g., "quarter note"), the consumed portion of one or more measures (e.g., half a measure), the number of metronome beats (e.g., three beats), etc. In an exemplary SNR event, the SNR onset time is indicated as occurring on the second beat of the fourth measure, and the SNR offset time is indicated as occurring on the fourth beat of the fourth measure. If necessary, additional MS logical data 143 can be used to convert the SNR offset time to a duration. For example, MS logical data 143 may indicate that the musical score is in "4 / 4" time, from which it can be derived that the exemplary SNR event is a half note. Additionally or alternatively, the SNR offset time can be indicated in the score-referenced MS logical data 143 as a duration. Referring to the same exemplary SNR event, MS logical data 143 may indicate that the SNR event has a duration of two beats or that the SNR event is a half note. In such cases, the SNR offset timing can be derived by adding the explicitly indicated duration to the SNR onset time, as needed. In either case, tempo can be used to convert score-referenced MS logical data 143 into real-time-referenced timing information. For example, additional MS logical data 143 may indicate a default tempo of 120 beats per minute, with a score-referenced half note consuming approximately one second of real time at the default tempo. The score-referenced definition of MS logical data 143 provides various features. For example, the real-time duration of all SNR events can be automatically adjusted with tempo changes.

[0020]

[0025] Alternatively, the MS logical data 143 can be referenced in real time. In one implementation, each SNR event is defined as having a start some milliseconds from the start of the song (i.e., SNR onset time) and an end some milliseconds from the start of the song (i.e., SNR offset time). In another implementation, each SNR event is defined as having a start some milliseconds from the start of the song (i.e., SNR onset time) and an end some milliseconds after the SNR onset time (i.e., SNR offset time). Regardless of whether the MS logical data 143 is score-referenced or real-time-referenced, the SNR offset time is defined as the associated end time of the SNR event or as the duration following the associated SNR onset time of the SNR event.

[0021]

[0026] As described herein, embodiments of rhythm evaluator subsystem 150 may evaluate a user's rhythmic performance only in conjunction with limited SNR event information. In some embodiments, rhythm evaluator subsystem 150 evaluates only with respect to SNR onset time and SNR offset time. In other embodiments, rhythm evaluator subsystem 150 evaluates further based on whether the SNR event is a note or a rest. In other embodiments, rhythm evaluator subsystem 150 evaluates further based on whether the SNR event is associated with a particular hand, finger, etc. For example, in these embodiments, rhythm evaluator subsystem 150 may evaluate a user's performance without considering pitch, dynamics, key signature, etc. Accordingly, embodiments of MS logical data 143 include only information useful for rhythmic evaluation by rhythm evaluator subsystem 150. In some such embodiments, display processor 125 generates a graphical representation of the musical score and SNR events based solely on MS visual data 145, without any processing of MS logical data 143. In other such embodiments, display processor 125 generates the graphical representation of the SNR event partially based on MS logical data 143 (e.g., determining the horizontal placement of notes and rests corresponding to the SNR event based on the SNR onset time, determining the note or rest type based on the SNR offset time, etc.), while the remaining graphical representation of the musical score and / or SNR event elements is generated from MS visual data 145. For example, MS logical data 143 is used to determine timing-related information of the SNR event, and MS visual data 145 is used to determine the pitch. Some embodiments of display processor 125 include a notation rules engine (not shown) that can derive additional information from MS visual data 145 and / or MS logical data 143 that can be used to generate the graphical representation of the musical score and / or SNR event elements.For example, the notation rules engine may determine whether notes have flags, whether notes are connected to each other across bar boundaries, whether note stems are pointing up or down, whether accidentals are indicated, etc.

[0022]

[0027] Embodiments of display interface 120 and display processor 125 are implemented in any suitable manner that supports the visual portion of the audiovisual output functionality described herein. The terms “visual” and “graphical” are used interchangeably herein. Display interface 120 can be implemented by any suitable display component, such as a tablet computer display, a smartphone display, a computer monitor, or the like. Display processor 125 can be implemented as any processor, portion of a processor, or group of processors capable of generating graphical output via display interface 120, as described herein. For example, display processor 125 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), or the like. In some implementations, a touchscreen interface integrates display interface 120 and haptic interface 110. As described above, display processor 125 can use MS visual data 145 (e.g., and further MS logical data 143) to generate a graphical representation of a musical score, including a graphical representation of a sequence of SNR events to be performed by a user.

[0023]

[0028] Embodiments of display processor 125 provide additional features. Some such features include generating a display output to graphically represent real-time progression through the musical score, such as by graphically indicating the current playback position. Other such features include generating a display output that graphically represents rating feedback. These and other features of display interface 120 and display processor 125 are described further below.

[0024]

[0029] Embodiments of audio interface 130 and audio processor 135 are implemented in any suitable manner that supports the auditory portion of the audiovisual output functionality described herein. The terms “auditory” and “audio” are used interchangeably herein. Audio interface 130 may be implemented by any suitable audio components, such as one or more audio transducers, speakers, headphones, etc. Audio processor 135 may be implemented as any processor, portion of a processor, or group of processors capable of generating audio output via audio interface 130 as described herein. In some embodiments, the audio representation of SNR events is dynamically generated from MS logic data 143. In some such embodiments, the audio representation of a sequence of SNR events indicates only the SNR onset time, such as by buzzing, vibrating, or clicking at each SNR onset time. In other such embodiments, the audio representation further indicates the SNR offset time of the SNR events (e.g., audio interface 130 beeps or vibrates for the duration between the SNR onset and SNR offset of each SNR event). In other such embodiments where MS logical data 143 includes additional audio information, the audio representation of the SNR events indicates further audio information such as pitch, dynamics, etc. In other embodiments, the audio representation of the SNR events is generated from recorded audio. For example, a sequence of SNR events is played by playing a digital audio file from MS data store 140 that stores a recording of a musical piece.

[0025]

[0030] Embodiments of audio processor 135 provide additional features. Some such features include generating additional audio during a user's performance, such as outputting a metronome and / or backing track to indicate tempo. Other such features include generating audio output to play back portions of a user's performance, for comparison with, for example, playback of an ideal version of a song, a recorded version of a song, a past performance by the user, etc. These and other features of audio interface 130 and audio processor 135 are described further below.

[0026]

[0031] As described above, embodiments of system 100 evaluate a user's rhythmic performance by obtaining haptic feedback from the user via haptic interface 110 while an audio-visual representation of the musical score is presented to the user via display interface 120 and audio interface 130. The audio-visual representation shows real-time progression through the musical score at a particular tempo, such as by sequentially highlighting each of the sequences of SNR events at their appropriate timing based on their notated rhythm and playback tempo. The haptic feedback corresponds to the user's interaction with haptic interface 110 representing the user's rhythmic performance of the sequences of SNR events of the musical score at the playback tempo. The haptic feedback is recorded as a sequence of haptic performance (HP) events.

[0027]

[0032] Haptic interface 110 may be implemented using any sensor capable of receiving HP events from a user, such that each HP event includes both an HP onset time and an HP offset time. In some implementations, haptic interface 110 is implemented by the touch sensor functionality of a touchscreen interface of a computing device. For example, when a user touches a touchscreen, haptic event processor 115 detects and records the touch event as a haptic event. In other implementations, haptic interface 110 is implemented by an input device integrated with the computing device. For example, a user may touch a trackpad, squeeze a force sensor, click an integrated mouse button, press an integrated keyboard key, press a key on an electronic piano keyboard, etc., and haptic event processor 115 detects and records the user interaction as a haptic event. In other implementations, haptic interface 110 is implemented by a peripheral input device that communicates with other parts of the computing device (e.g., via a wired or wireless connection). For example, a user may click a button on a peripheral mouse or keyboard, grasp a peripheral haptic glove, press a key on an external electronic piano keyboard, etc., and the haptic event processor 115 detects and records the user interactions as haptic events. While the embodiments described herein generally describe haptic interfaces 110 that interface with a user's hands or fingers, other types of haptic interfaces 110 may also be used. For example, haptic interface 110 may interface with a user's feet, a user's breath, etc.

[0028]

[0033] In some implementations, information received from haptic interface 110 can be supplemented with additional input received via other inputs. For example, in addition to (e.g., simultaneously with) interacting with haptic interface 110, a user can also clap, snap, whistle, sing, tap their feet, etc. Such additional interactions can be detected and processed via audio and / or video input (e.g., via a microphone, camera, etc.). In some cases, such additional interactions can be used to set an initial tempo for a performance, to receive additional commands during the user's performance, or to facilitate verification of haptic feedback (e.g., to improve reliability determination).

[0029]

[0034] In any of the above or other implementations, the haptic interface 110 is implemented to allow a user to control the HP onset time and the HP offset time. In some such implementations, the user initiates engagement with the haptic interface 110 to establish the HP onset time of the HP event, maintains engagement with the haptic interface 110 for some duration, and releases engagement from the haptic interface 110 to establish the HP offset time. For example, the user places a finger on the touchscreen interface (e.g., presses a key or button, etc.) to define the HP onset time and removes the finger from the touchscreen interface (e.g., releases a key or button, etc.) to define the HP offset time. Other implementations may provide alternative methods of indicating the HP offset time. For example, after engaging with haptic interface 110 to establish the HP onset time of an HP event, the user can establish the HP offset time by re-engaging with haptic interface 110 (i.e., a first touch or click indicates the start of the HP event and a second touch or click indicates the end of the HP event), by engaging with a different portion of haptic interface 110 or a different haptic interface 110, etc. In some embodiments, the HP onset time and HP offset time are defined in opposite ways. For example, haptic interface 110 and / or haptic event processor 115 may be configured such that disengagement from haptic interface 110 indicates the HP onset time and engagement with haptic interface 110 indicates the corresponding HP offset time.

[0030]

[0035] In some embodiments, the haptic interface 110 and the haptic event processor 115 are configured to capture multiple simultaneous tracks of HP events. As one example context, a piano student may wish to learn a musical piece with notated right-hand and left-hand parts to be played simultaneously at different rhythms. As another example context, a piano student may wish to practice a rhythmically complex passage of a piece of music in which different fingers are playing at different rhythms (e.g., some fingers hold certain notes while other fingers of the same hand change to different notes). In some implementations, a single haptic interface 110 can receive multiple tracks of HP events simultaneously. For example, a multi-touch touchscreen interface, a multi-key keyboard, a multi-button mouse, etc., can receive multiple simultaneous haptic interactions. In other implementations, multiple haptic interfaces 110 are used simultaneously to receive multiple simultaneous haptic interactions.

[0031]

[0036] In some embodiments, additional haptic information can be detected and / or haptic feedback can be used for additional purposes. As described above, some embodiments of the rhythm evaluator subsystem 150 evaluate user performance based solely on onset and offset timing, without consideration of pitch, dynamics, key signature, etc. In other embodiments, additional haptic interactions can be detected. For example, a touchscreen or other haptic interface 110 may detect how hard a user is pressing, or a haptic glove, three-dimensional mouse or controller, or other haptic interface 110 may detect velocity, acceleration, orientation, three-dimensional gestures, etc. In some implementations, such additional haptic interactions can be used by the rhythm evaluator subsystem 150 to evaluate dynamics and / or other characteristics of the user's performance. In some embodiments, the above-described HP events and / or such additional haptic interactions can be used for purposes other than rhythm evaluation, such as user interface navigation, playback control, etc. For example, a user may tap haptic interface 110 at a particular rate before the start of a performance to set a performance tempo, or a user may tap each finger on haptic interface 110 before a performance to calibrate haptic interface 110 to finger positions, sensitivity levels, etc.

[0032]

[0037] User interactions with haptic interface 110 (e.g., one at a time or multiple simultaneously) are detected by haptic event processor 115, which generates HP event data 117. In some implementations, HP event data 117 represents a sequence of HP events, each having an associated HP onset time and an associated HP offset time. In other implementations, HP event data 117 represents a sequence of HP onset times and HP offset times from which the sequence of HP events can be derived. For example, if only a single track of HP events is received, each sequential pair of HP onset time and HP offset time can be treated as defining the start and end, respectively, of the associated HP event.

[0033]

[0038] Embodiments of the haptic event processor 115 can use various approaches to recognize multiple simultaneous tracks of HP events. In some implementations in which multiple haptic interfaces 110 are used, user interactions with each haptic interface 110 are labeled based on its originating haptic interface 110 (e.g., based on the logical port through which the haptic data is received). In other implementations in which a single haptic interface 110 is used, the haptic interface 110 provides additional data that can be used by the haptic event processor 115 to separate tracks. In some such implementations in which haptic data comes from a multi-key or multi-button haptic interface 110, the haptic event processor 115 can group user interactions with keys or buttons to separate simultaneous HP events. For example, on a typing keyboard, a user presses the “S” key at time T0, presses the “K” key at time T1, releases the “K” key at time T2, and releases the “S” key at time T3. If only a single sequence is maintained (HP onset at T0, HP onset at T1, HP offset at T2, HP offset at T3), there is no way to know whether the HP offset at T2 is associated with the first HP onset or the second HP onset. By also recording the associated keys, haptic event processor 115 can group the HP onset at T0 with the HP offset at T3 as both including the "S" key, and haptic event processor 115 can group the HP onset at T1 with the HP offset at T2 as both including the "K" key. In other such implementations where haptic data comes from a multi-touch interface (e.g., a multi-touch touchscreen or trackpad), haptic event processor 115 can group HP onset and HP offset times to separate simultaneous HP events based on the location or region of the multi-touch interface where each user interaction occurs.As one example, a user may use both hands and / or multiple fingers simultaneously to interact with the touchscreen, thereby essentially generating multiple simultaneous interactions with different locations on the touchscreen, and the haptic event processor 115 may group these interactions by location to determine which HP onset time and which HP offset time to pair. As another example, separate regions of the touchscreen are pre-designated (e.g., labeled, color-coded, etc.) to receive respective tracks of HP events (e.g., right-hand region and left-hand region), and the haptic event processor 115 is configured to process interactions from each region in relation to its pre-designated track.

[0034]

[0039] In some embodiments, the HP onset time always indicates the start of a user interaction with haptic interface 110, and the HP offset time always indicates the end of a user interaction with haptic interface 110. In some such embodiments, a "note" event (whether an SNR event or an HP event) is identified as an onset time followed by a corresponding offset time, and a "rest" event (whether an SNR event or an HP event) is identified as an offset time followed by a corresponding onset time. In other embodiments, a note HP event is characterized by an onset time followed by an offset time, and a rest event is characterized by an offset time followed by a corresponding onset time.

[0035]

[0040] Embodiments send the generated HP event data 117 to an HP event store 153. For example, the haptic event processor 115 can include a local buffer that records detected haptic events, and the haptic event processor 115 outputs the HP event data 117 as one or more tracks of a sequence of HP events and / or a sequence of HP onset times and HP offset times. In some implementations, the HP event store 153 is implemented in the device I / O subsystem 105, such as integrated with the haptic event processor 115. In other implementations, the HP event store 153 is implemented in the rhythm evaluator subsystem 150, as shown. In other implementations, the HP event store 153 is implemented as part of a storage subsystem (not shown) that includes the HP event store 153 and the MS data store 140. The HP event store 153 may be implemented using any suitable data storage, such as solid-state storage, hard disk storage, removable media storage, network (e.g., local network, cloud-based, etc.) storage, etc. Furthermore, the HP event data 117 may be stored by the HP event store 153 in any suitable manner, such as a flat file data structure, a hierarchical data structure, or the like.

[0036]

[0041] An embodiment of rhythm evaluator subsystem 150 evaluates a user's performance (i.e., HP event data 117 acquired and stored in HP event store 153) based on how accurately the user's performed rhythm corresponds to the rhythm represented by the sequence of SNR events notated in the musical score. An embodiment of rhythm evaluator subsystem 150 includes evaluation engine 155 and feedback engine 157. Evaluation engine 155 generates evaluation data based on comparing performance data from HP event store 153 with SNR data represented by MS logic data 143. The evaluation data is used by feedback engine 157 to generate feedback data 159. Feedback data 159 can include micro-level feedback and macro-level feedback. Feedback data 159 can be used by display processor 125 to generate graphical performance feedback that is output to the user via display interface 120.

[0037]

[0042] In some embodiments, the evaluation engine 155 performs its evaluation by iterating through each of the sequence of SNR events. For each SNR event, the evaluation engine 155 finds a matching HP event (i.e., an HP event representing a user's attempt to perform the corresponding SNR event) and attempts to determine how accurately the matching HP event matches the rhythm of the corresponding SNR event (e.g., its SNR onset time and / or SNR offset time). The timing of the HP events is recorded with reference to a performance time reference. For example, each HP onset time and HP offset time is represented by a timestamp that is a certain temporal distance (e.g., in milliseconds) from the beginning of the performance. The performance time reference may be the same as the real-time reference described above, or any other suitable time reference. As described above, some embodiments include score-referenced MS logical data 143. Such embodiments can convert the score-referenced MS logical data 143 to a performance time reference. For example, score-referenced MS logical data 143 is converted to real-time referenced MS logical data 143, and if necessary, the real-time referenced MS logical data 143 is further time-aligned to a performance time reference (or vice versa).

[0038]

[0043] The evaluation engine 155 can obtain a set of evaluation criteria for each SNR event. The evaluation criteria can be formulated as processor-readable evaluation functions, such as if-then statements, matching functions, etc. Some or all of the evaluation criteria can be stored as part of the MS logic data 143 for the SNR event, and / or some or all of the evaluation criteria can be derived based on predetermined thresholds and / or rules. One type of evaluation criterion is a “valid match window.” The valid match window is a time window referenced to the SNR onset time of each SNR event. In some implementations, the valid match window is symmetric with respect to the SNR onset time, such as a 100-millisecond window extending 50 milliseconds on either side of the SNR onset time. In other implementations, the valid match window is asymmetric, such as a 100-millisecond window beginning 25 milliseconds before the SNR onset time and ending 75 milliseconds after the SNR onset time. In some implementations, the valid match window is a predetermined fixed size referenced to real time. For example, the effective match window is always 100 milliseconds and is always symmetric about the SNR onset time of all SNR events. In some implementations, the effective match window is score-referenced and rule-based. In one such implementation, the effective match window is defined as a function of beats (e.g., a ratio or percentage, or some formulaic relationship). For example, the effective match window shrinks with faster tempos. In another such implementation, the effective match window is defined as a function of the user's skill level and / or the difficulty of the musical score. For example, the effective match window is more tolerant of less advanced players and / or simpler scores.

[0039]

[0044] As described above, for each SNR event, the rating engine 155 can find a matching HP event (representing the user's performance for the SNR event). An embodiment of the rating engine 155 can search the HP events in the HP event store 153 to determine whether any of the retrieved HP events are "matching events," meaning that the HP event has an HP onset time that falls within the valid match window of the SNR event. In some implementations, if multiple HP events are found to have HP onset times within the valid match window, the rating engine 155 selects the HP event with the HP onset time closest to the SNR onset time as the matching event. Some embodiments iterate through all SNR events in the MS data store 140 to find matching HP events. Other embodiments may ignore some of the SNR events in the MS data store 140. For example, if a musical score includes a particular type of musical grouping (e.g., quintuplets) and / or a particular idiom (e.g., tempo rubato passages), the SNR events associated with these portions of the musical score may be ignored or treated differently.

[0040]

[0045] For each evaluated SNR event, the evaluation engine 155 either finds a matching HP event or does not. If no matching HP event is identified (i.e., no HP event with an HP onset time exists within the valid match window of the SNR event), the SNR event can be tagged as a "skipped event." If a matching HP event is identified (i.e., an HP event with an HP onset time exists within the valid match window of the SNR event), the matching HP event can be associated with the SNR event. Such tagging can be performed in any suitable manner, such as by setting a flag associated with the SNR event, storing a label associated with the SNR event, etc. In some embodiments, the rhythm evaluator subsystem 150 includes an evaluation data store 156 for storing evaluation results. For example, additional storage can be used to store an instance of each evaluated SNR event along with its generated evaluation data (e.g., tag, label, etc.) and / or store the evaluation data along with a pointer or other identifier linking the evaluation data to corresponding ones of the SNR events stored in the MS data store 140.

[0041]

[0046] If a matching HP event is identified (i.e., there is an HP event with an HP onset time within the valid match window of the SNR event), the matching HP event can be associated with the SNR event. Each matching event (or matching HP event) is one of the HP events determined to correspond in time to one of the SNR events. Such association can be performed by storing a pointer between the matching HP event and the evaluated SNR event, or in any other suitable manner. Each matching event is one of the HP events determined to correspond in time to one of the SNR events.

[0042]

[0047] For each matching HP event, the evaluation engine 155 can then evaluate the rhythmic compatibility between the matching HP event and its temporally corresponding SNR event. The rhythmic compatibility can be based on one or more of at least three rhythmicities of the SNR event: onset compatibility, such as between the HP onset time and the SNR onset time; offset compatibility, such as between the HP offset time and the SNR offset time; and / or duration compatibility, such as between the duration of the HP event (i.e., the temporal distance between the HP onset time and the HP offset time) and the duration of the SNR event (i.e., the temporal distance between the SNR onset time and the SNR offset time). In some embodiments, the rhythmic compatibility between the matching HP event and its temporally corresponding SNR event is expressed as a score functionally related to the temporal distance between the characteristics. For example, a high onset compatibility score indicates a short temporal distance between the HP onset time and the SNR onset time. Scoring can vary for different characteristics. In other embodiments, the rhythmic compatibility between the matching HP event and its temporally corresponding SNR event is expressed categorically based on one or more predetermined thresholds. For example, the onset suitability is determined to be "good" or "bad" (or "high," "medium," and "low," etc.) by determining whether the temporal distance between the HP onset time and the SNR onset time is greater than or less than one or more predetermined thresholds. The one or more predetermined thresholds may be different for different characteristics.

[0043]

[0048] As described above, the evaluation engine 155 can obtain a set of evaluation criteria for each SNR event. The evaluation criteria can include setting a valid window, an evaluation threshold, scoring, and / or other parameters for evaluating rhythmic conformance. In some embodiments, the evaluation criteria are generated based on predetermined fixed settings. For example, as described above, the valid match window can be defined as a fixed time window relative to the SNR onset time. In other embodiments, some or all of the evaluation criteria are rule-based. Such rule-based evaluation criteria can be used to adjust the evaluation of rhythmic conformance based on tempo, the user's skill level, the difficulty of the musical score or passage, the SNR event type, and / or other additional information. For example, the valid match window can be adjusted so that matching HP events are identified more or less strictly according to tempo, note type, etc.

[0044]

[0049] As another example, the evaluation criteria may be set so that the rhythmic compatibility of note SNR events is evaluated based on stricter onset compatibility and looser offset compatibility, and the rhythmic compatibility of rest SNR events is evaluated based on stricter onset compatibility and stricter offset compatibility. As another example, the evaluation criteria may be used to set a valid offset window. The valid offset window may define a window for evaluating offset compatibility for the HP offset time of a matching HP event and / or for evaluating duration compatibility for the duration of the HP event. For example, the valid offset window may determine an appropriate HP offset time and / or HP event duration as a temporal distance from the HP onset time based on whether the corresponding SNR event is staccato, legato, rest, etc.

[0045]

[0050] Evaluation criteria may also be formulated and applied by the rhythm evaluator subsystem 150 to handle special cases. One category of special cases includes musically related groupings of SNR events. One example of a musically related grouping is a chord. In some cases, the rhythm evaluator subsystem 150 evaluates a chord as a single SNR event. For example, a chord can be stored in a single set of MS logical data 143 or treated as a single event with a single SNR onset time, a single SNR offset time, a single duration, etc. In other cases, such as when using the multi-touch haptic interface 110 to obtain multiple simultaneous tracks of HP events representing the fingers of multiple hands, etc., each note of a chord can be treated as a separate SNR event. For example, in some musical compositions, one or more notes of a chord are held and other notes are released (e.g., notated by using one stem direction for the held note or notes and another stem direction for the moving note or notes).

[0046]

[0051] Another example of a musically related grouping is a "quintuplet" (a grouping of five sixteenth notes to be performed within the span of one quarter note), or any other grouping of several notes to be played over a defined number of beats. One implementation can treat each note of the grouping as any other note, with each note having its own SNR onset time and SNR offset (and / or SNR event duration) so that it is evenly spaced and consumes exactly each fifth note of the quarter note duration. In such an implementation, each note can be evaluated for rhythmic compatibility, such as by evaluation engine 155, which looks for matching HP events. Other implementations can treat the entire grouping as an SNR event (e.g., instead of or in addition to storing each component of the grouping as an SNR event). In one such implementation, the evaluation engine 155 can locate an appropriate set of constituent HP events (e.g., five-note HP events) as matching events by determining whether the set of constituent HP events begins within the valid match window of the grouped SNR events and all fall within the valid offset window of the grouped SNR events (e.g., corresponding to the total duration of a quintet). Similar approaches can be applied to other types of musically related groupings, such as the treatment of grace notes, trills, mordents, glissandi, etc.

[0047]

[0052] Another category of special cases involves expressive notations. One example is a passage marked "tempo rubato" in the score, indicating that it should be played freely. In some cases (e.g., for some genres, composers, etc.), such a tempo rubato passage may indicate that the performer is completely free from rhythmic constraints. In some such cases, embodiments may configure evaluation criteria to ignore SNR events that fall within the tempo rubato section when performing rhythmic evaluation. For example, these SNR events may be specially tagged or labeled. In other such cases, the evaluation engine 155 uses pattern matching to determine which set of HP events represents the user's performance of the tempo rubato passage in order to correct subsequent timing. Such pattern matching may include counting the number of HP events corresponding to the number of SNR events in the passage, attempting to match the relative durations of the HP events to the different durations of the SNR events in the passage, etc. By using such a pattern matching approach, the evaluation engine 155 can shift the performance time basis after the passage (i.e., shift the timing of the HP event after the tempo rubato passage) so that the HP event can be properly matched to the temporally corresponding SNR event after the tempo rubato passage, regardless of how long the performer spent performing the passage.

[0048]

[0053] In other cases, certain rhythmic constraints are anticipated even if a passage is marked as rubato. For example, performing a tempo rubato section of a particular piece may involve strict adherence to rhythm with the left hand while allowing more rhythmic freedom with the right hand. As another example, some composers or genres intend tempo rubato passages to be performed for a fixed overall duration (e.g., consuming a total number of seconds, beats, or measures), even if individual notes within the passage may be accelerated or decelerated in personal and expressive ways. In such cases, embodiments may set stricter evaluation criteria for passage-level rhythmic evaluation, while setting looser or no evaluation criteria for note- or rest-level rhythmic evaluation. In one such case, evaluation criteria are defined to enable the evaluation engine 155 to search for a set of matching HP events, where the first of the set of matching HP events has a good onset fit with the SNR onset time of the first SNR event in the passage, the last of the set of matching HP events has a good offset fit with the SNR offset time of the last SNR event in the passage, and the number of HP events in the set of HP events is within a predetermined threshold amount (e.g., number, percentage, etc.) of the total number of SNR events in the passage. In another such case, the evaluation criteria include additional note-level or rest-level definitions. For example, the evaluation criteria may look for the onset fit of each SNR event in the passage to determine whether any of the matching HP events were performed outside a predetermined tolerance (e.g., the HP onset time is too long before or after the corresponding SNR onset time). Similar approaches can be applied to other types of expression notation, such as fermata handling.

[0049]

[0054] Based on the above, evaluation engine 155 uses the evaluation criteria to generate evaluation data indicative of rhythmic compatibility between at least the user's performance and a sequence of SNR events (e.g., all of the SNR events of a musical score, or an evaluated portion of the SNR events). In some embodiments, the evaluation data is stored in evaluation data store 156. An embodiment of feedback engine 157 can then use the evaluation data (e.g., from evaluation data store 156) to generate and output feedback data 159. In some embodiments, feedback data 159 is generated such that it is usable by display processor 125 to generate graphical performance feedback that is output to the user via display interface 120. In some embodiments, feedback engine 157 generates feedback data 159 that is further usable by audio processor 135 to generate audible performance feedback that is output to the user via audio interface 130.

[0050]

[0055] In some embodiments, the feedback data 159 is micro-level feedback indicating per-event rhythmic compatibility as a graphic overlay on the musical score. The term “graphic overlay” is used herein generally to include any type of simultaneous graphic display that provides a visual juxtaposition between a graphic feedback element and the graphic score element to which the feedback applies. For example, such graphic overlays may include semi-transparently displaying the graphic feedback element over the graphic score element, recoloring the graphic score element to imply the graphic feedback element (e.g., changing the line thickness, etc.), adding text or an image representing the graphic feedback element along with the graphic score element, etc. In one implementation, SNR events are displayed on the musical score in a first color to indicate skipped events (i.e., no matching HP event was found), a second color to indicate well-performed events (e.g., a matching HP event was found and the matching HP event has a good onset, offset, and / or duration compatibility), and a third color to indicate poorly-performed events (e.g., a matching HP event was found but the onset, offset, and / or duration of the matching HP event was poorly compatible). In another implementation, a first color is used to highlight the portion of the musical score where the SNR event should be performed (e.g., from the SNR onset time to the SNR offset time of the SNR event for a particular note), a second color is used to highlight the portion of the musical score where the matching HP event was performed well, and a third color is used to highlight the portion of the musical score where the matching HP event was performed poorly. In another implementation, each SNR event is labeled with a score indicating the assessed rhythmic compatibility for that SNR event. In another implementation, colors or other graphical elements are further used to overlay past performance data from the user or other performers.

[0051]

[0056] In other embodiments, the feedback data 159 is macro-level feedback indicating rhythmic conformance for each performance. The macro-level feedback may be presented as a graphic overlay on the musical score, in a separate portion of the display, or in any suitable manner. In some such embodiments, the macro-level feedback indicates an overall performance score (e.g., in textual form and / or any other suitable format). For example, the feedback data 159 indicates that the performance had an overall rhythmic conformance of 82%. In other such embodiments, the feedback engine 157 performs one or more statistical analyses on the evaluation data to look for patterns or trends, and the feedback data 159 indicates the results of those analyses. One such analysis indicates a pattern of performance across passages and / or sections of the performance. For example, the feedback data 159 indicates that the user's performance was good (i.e., had high rhythmic conformance) in measures 1-10, very poor in measures 11-13, and moderately good in the remainder of the performance. Another such analysis matches performance patterns to predetermined performance categories. As one example, the feedback data 159 indicates poor performance on rests rather than notes (e.g., the analysis compares the rhythmic correspondence of rest SNR events with non-SNR events). As another example, the feedback data 159 indicates that there was an overall tendency to play before or after the beat (e.g., the analysis finds that there is a statistical tendency for HP onset times to be earlier than or later than the SNR onset times). As another example, the feedback data 159 indicates that there was a misunderstanding of certain musical notations (e.g., the analysis finds that two HP events are present even when two notes are tied together, indicating a misunderstanding of cohesion; HP events consistent with triplets all appear to have eighth-note durations and intervals, indicating a misunderstanding of triplets, etc.).As another example, feedback data 159 may indicate that a performer does not appear to be attempting a portion of a musical score (e.g., analysis may reveal that HP events are missing throughout one hand of a two-handed piece, or throughout a section of a piece). As another example, feedback data 159 may indicate that a performer appears to be struggling with a particular type of passage (e.g., analysis may reveal that rhythmic correspondences tend to be higher during fast or slow sections of a piece).

[0052]

[0057] The above assumes that rhythmic evaluation is performed from the SNR event direction, such as by attempting to match the user's performed HP events to SNR events in the score. For example, each SNR is evaluated to find a matching HP event, or the event is considered to be skipped. Some embodiments of the rhythm evaluator subsystem 150 perform further and / or alternative evaluation from the HP event direction. In some such embodiments, the HP events are analyzed to look for potential interface errors. For example, a glitch in the haptic interface 110 or slight movements of the user's hand or fingers on the haptic interface 110 may cause certain types of haptic event processor 115 to record multiple HP events, even if only a single HP event was intended by the performer. Some implementations can address such cases in preprocessing. The haptic event processor 115 can detect when the HP offset time of one HP event and the HP onset time of the next HP event are closer than a predetermined threshold, indicating that the user is unlikely to have intended separate events, and the haptic event processor 115 can treat these events as a single HP event (e.g., by generating a single event in the HP event data 117 and removing data for intermediate offset and onset times). Other implementations can address such cases in post-processing. For example, the evaluation engine 155 can detect a group of consecutive HP events (e.g., two, three, etc.), where the first HP event has a high onset match with the SNR onset time of the SNR event, the last HP event has a high offset match with the SNR offset time of the same SNR event, and the consecutive HP event groups have a minimum interval between them. Removal of unintended HP event data in these and other cases can be referred to as “de-duplication.”

[0053]

[0058] In other such embodiments, HP events are analyzed (e.g., by the rating engine 155) to rate unmatched (non-deduplicated) HP events. As one example, there may be more HP events than SNR events, but some or all of the unrelated HP events appear to have been intentionally performed by the user. Some embodiments of the rating engine 155 analyze those unrelated HP events to determine whether they represent a misunderstanding of certain notational conventions (e.g., note cohesion), whether they suggest an error in processing haptic information, whether the user appears to have been improvising or riffing for a moment, etc. As another example, the rating engine 155 may determine that the number of SNR events identified as skipped events is similar to the number of HP events not identified as matching events (e.g., an entire section of the HP event may have been performed correctly relative to itself but was somehow time-shifted relative to the SNR events). Embodiments of the rating engine 155 can identify such cases and / or attempt to remedy them (e.g., by attempting to map unmatched HP events to missed SNR events using a moving window or other pattern matching technique).

[0054]

[0059] 2 illustrates another exemplary haptic rhythm training system 200 implemented in a cloud-based environment in accordance with various embodiments described herein. The system 200 includes one or more user devices 210 (only a single instance of each is shown to avoid overcomplicating the figure) communicating with one or more remote servers 230 via one or more communication networks 240. Each user device 210 may implement a respective instance of a device I / O subsystem 105, a thin client 215, and a network interface 220. The one or more remote servers 230 implement a rhythm evaluator subsystem 150 and a remote storage subsystem 235. The remote storage subsystem 235 may include some or all of the MS data store 140, the HP event store 153, and the evaluation data store 156. For example, the remote storage subsystem 235 may be used to maintain records of historical performance data.

[0055]

[0060] For example, during operation, a user uses the thin client 215 of the user device 210 to access applications that provide the functionality of the rhythm evaluator subsystem 150 by communicating over one or more networks 240. The operation of the thin client 215 may include communication of various types of data. For example, MS visual data 145 and MS logical data 143 are received by the user device 210 from the remote storage subsystem 235 of one or more remote servers 230, HP event data 117 is transmitted from the user device 210 back to the rhythm evaluator subsystem 150 of one or more remote servers 230, and feedback data 159 is received by the user device 210 from the rhythm evaluator subsystem 150 of one or more remote servers 230. Communication between the user device 210 and the one or more remote servers 230 over the one or more networks 240 is facilitated by the network interface 220. For example, the one or more networks 240 may include any suitable wired or wireless communication links with any one or more public and / or private networks, local and / or remote networks, etc., and the network interface 220 implements (or facilitates the use of) any associated ports, protocols, etc.

[0056]

[0061] 3A-3C illustrate example screenshots 300 from an example application for implementing features of a haptic rhythm training system according to various embodiments described herein. Referring to FIG. 3A, first screenshot 300a illustrates a graphical performance interface having several regions or portions. FIGS. 3A-3C contemplate a computing environment in which a touchscreen display implements display interface 120 for graphical output and haptic interface 110 for tactile input. Static or dynamic graphical output to display interface 120 is generated by display processor 125, and any haptic interaction with haptic interface 110 (e.g., a user tapping or holding a finger or hand on the touchscreen) is detected by haptic event processor 115 and converted into HP event data 117.

[0057]

[0062] As shown, a first portion of the graphical performance interface (first interface region 310) displays a musical score with a musical notation representation of the SNR events. In the illustrated example, first interface region 310 shows a simple piano arrangement of a portion of Beethoven's "Ode to Joy" from Symphony No. 9. The displayed musical score includes a grand staff with treble and bass clefs, notations for notes and rests (corresponding to the sequence of SNR events), notations for time signatures, notations for bar boundaries, notations for right and left hand fingerings, etc.

[0058]

[0063] A second portion of the graphical performance interface (second interface area 320) is for haptic input. As described herein, embodiments may implement second interface area 320 in different ways. In some implementations, only a single track of HP events is received, such that only a single haptic interface area is required. Some such implementations may include a single designated area, such as a single area with a graphically defined boundary, for accessing haptic interactions that are detected as HP events. For example, haptic event processor 115 is configured such that only user interactions within the designated area of ​​haptic interface 110 are used to generate HP event data 117. Other such implementations do not graphically designate any particular area, and any haptic interaction with any portion of haptic interface 110 is detected as an HP event by haptic event processor 115.

[0059]

[0064] In other implementations, multiple tracks of HP events are received simultaneously, requiring multiple haptic interface sub-regions. Some such implementations include explicit graphical definition of the multiple sub-regions, such as by displaying graphically defined boundaries. In the illustrated example, the second interface region 320 includes two explicitly designated sub-regions: one for left-hand tactile input labeled "L" and the other for right-hand tactile input labeled "R." Any tactile interaction within the first sub-region of the haptic interface 110 is detected by the haptic event processor 115 as a left-hand HP event, and any tactile interaction within the second sub-region of the haptic interface 110 is detected by the haptic event processor 115 as a right-hand HP event. Other implementations may specify any suitable number of sub-regions. For example, four sub-regions may be specified to practice harmony rhythms with four-part movements, or ten sub-regions may be specified to practice using all ten fingers. Implementations may specify sub-regions in any suitable manner. For example, one of the two sub-regions may be graphically designated as the left third of the touchscreen, and the other of the two sub-regions may be graphically designated as the right third of the touchscreen, or ten sub-regions may be graphically designated to look like two sets of five fingers or two sets of five circles. Other such implementations do not include explicit graphical designation of sub-regions. Rather, the haptic event processor 115 may group haptic inputs by location such that detected interactions with the haptic interface 110 that occur within a threshold distance of each other are grouped into a single track of HP events. For example, during a particular practice session, the user's right thumb may always contact the touchscreen within a 9-inch square area near the bottom-right region of the display, and the user's left thumb may always contact the touchscreen within a 6-inch square area near the center-left region of the display; therefore, the haptic event processor 115 may identify the interactions with the right and left thumbs as separate, simultaneous tracks of HP events.

[0060]

[0065] In some embodiments, the first interface region 310 and the second interface region 320 do not completely overlap, and no portion of the second interface region 320 overlaps any portion of the first interface region 310. In other embodiments, the first interface region 310 and the second interface region 320 partially overlap. In other embodiments, the first interface region 310 and the second interface region 320 completely overlap. For example, the first interface region 310 and the second interface region 320 consume the same area, and the second interface region 320 is completely within the first interface region 310, or the first interface region 310 is completely within the second interface region 320.

[0061]

[0066] In some embodiments, the first interface area 310 is an output-only area, and haptic interactions with the first interface area 310 are ignored by the haptic event processor 115. In one such embodiment, only haptic interactions in the second interface area 320 (separate from the first interface area 310) are detected by the haptic event processor 115. In another such embodiment, the graphical performance interface includes one or more additional areas other than the first interface area 310 and the second interface area 320 where haptic interactions are detected and processed but not for generating HP event data 117. For example, other haptic interactions in other areas of the graphical performance interface may be detected for interface navigation or control, such as to pause or resume playback, to end a practice, to access one or more menus, to switch applications, etc. In other embodiments, haptic interactions with the first interface area 310 are not used for HP event generation but are still detected by the haptic event processor 115 to support other functionality. For example, tapping the first interface area 310 may pause or resume playback, and double-tapping the first interface area 310 may open a menu.

[0062]

[0067] In operation, as described herein, the output in the first interface area 310 is used to provide at least a graphical representation of real-time progression through the musical score at the tempo. The real-time progression may be indicated by a cursor or other graphical element 315 indicating a current playback position 315 on the score mapped to the current playback time at the tempo. As shown, the graphical element may include a box highlighting one of the currently playing SNR events as graphically displayed on the musical score. Alternatively, the same type of highlight box may indicate the current beat of playback (e.g., all SNR events that fall within that beat are within the highlighted box). Additionally or alternatively, the graphical element may include a vertical line moving horizontally across the measures of the score at a rate corresponding to the playback tempo. During the output of the graphical representation in the first interface area 310, HP events are captured in the second interface area 320. The HP events represent the user's rhythmic performance of the musical score at the tempo as it is being played back at the tempo.

[0063]

[0068] Referring to FIG. 3B , a second screenshot 300b shows a graphical performance after several user haptic interactions have been acquired (e.g., during a practice performance). For example, the current playback position 315 is indicated as being on the first beat of the last bar of the score. During the user's performance, real-time graphical feedback can be provided to indicate the HP event data 117 being generated by the haptic interactions the user has performed. In the illustrated example, highlights (e.g., a color different from the color of the current playback position 315 indication) indicate the acquired HP events overlaid on the graphical representation of the SNR events. Each highlight can show a graphical representation of a respective HP onset time 330 (i.e., corresponding to the time the user began interacting with the haptic interface 110), a graphical representation of a respective HP offset time 331 (i.e., corresponding to the time the user stopped interacting with the haptic interface 110), and / or a graphical representation of a respective HP event duration 332 (i.e., corresponding to the amount of time the user maintained interaction with the haptic interface 110). The performer can understand, for example, that not all quarter notes are held for the same amount of time. The figure also shows that some SNR events are not overlaid on the HP event data 117, making it appear as though these SNR events were not performed by the user.

[0064]

[0069] Referring to FIG. 3C , a third screenshot 300c shows a graphical performance after the performance has been completed and evaluated by the rhythm evaluator subsystem 150. The current playback position 315 is not shown because the musical score is no longer being played. Visual feedback 340 is provided to graphically map the evaluation data (generated by the rhythm evaluator subsystem 150) to a musical notation representation of the SNR events on the musical score. In the illustrated example, a first color of highlighting is used to indicate skipped events for SNR events detected as not having temporally corresponding HP event data, and at least a second color of highlighting is used to indicate executed events for SNR events detected as having temporally corresponding HP event data (e.g., the highlighting can further distinguish between successful and unsuccessful events).

[0065]

[0070] As described herein, some embodiments can provide both micro-level and macro-level feedback. In the illustrated example, visual feedback 340 is used to provide micro-level feedback. For example, visual feedback 340 indicates a rhythmic performance evaluation for individual SNR elements (e.g., and / or groups of SNR elements). In some embodiments, feedback area 350 is provided to display macro-level feedback. In the illustrated example, feedback area 350 replaces second interface area 320. For example, rhythm evaluator subsystem 150 may determine that the user did not perform a section of the score (e.g., there is a section with multiple consecutive missed events) and generate corresponding macro-level feedback such as, "It looks like you stopped playing in the middle of the song. Try again and give it your all!" Similarly, rhythm evaluator subsystem 150 may determine that the user did not perform any of the score (e.g., no HP events detected throughout the performance) and generate corresponding macro-level feedback such as, "It looks like you decided not to perform!"

[0066]

[0071] Some embodiments include additional types of interfaces for providing historical feedback and / or other information to the user. For example, FIGS. 6A and 6B show an example screenshot 600 from an example application for implementing the historical user feedback functionality of a haptic rhythmic training system according to various embodiments described herein. In the illustrated example, the display interface includes a historical feedback control area 610 including a slider bar. The slider bar indicates past performance times. By sliding to different positions on the slider bar, the user can access feedback from those respective performance times. For example, screenshot 600a in FIG. 6A shows a performance at 09:54 on August 12, while screenshot 600b in FIG. 6B shows a performance four minutes later. Each performance resulted in different feedback. Such historical feedback data can be stored in local storage (e.g., in the system of FIG. 1), remote storage (e.g., in the system of FIG. 2), or any suitable manner. Such past performance data can be displayed to the user in alternative ways. Some implementations overlay historical data from multiple performances. Some implementations generate trend data (e.g., statistics and / or other information) based on past performances. Some such implementations generate and provide macro-level feedback based on trend data over past performance, and some such implementations suggest additional practice based on such trend data.

[0067]

[0072] Embodiments of the haptic rhythmic training system, or components thereof, can be implemented on and / or incorporated into one or more computer systems, as shown in FIG. 4. FIG. 4 provides a schematic diagram of one embodiment of a computer system 400 that can implement various system components and / or perform various steps of methods provided by various embodiments. Note that FIG. 4 is only meant to provide a generalized description of various components, any or all of which may be suitably utilized. Thus, FIG. 4 broadly illustrates how individual system elements can be implemented in a relatively separate or relatively more integrated manner.

[0068]

[0073] Computer system 400 is shown including hardware elements that may be electrically coupled (or may otherwise communicate, as needed) via bus 405. The hardware elements may include one or more processors 410, including, but not limited to, one or more general-purpose processors and / or one or more special-purpose processors (digital signal processing chips, graphics acceleration processors, video decoders, etc.). As shown, some embodiments include device I / O subsystem 105, which may include one or more I / O devices 415 and one or more I / O processors 417. As described herein, I / O device 415 may include haptic interface 110, display interface 120, and audio interface 130. I / O processor 417 may include haptic event processor 115, display processor 125, and audio processor 135. Additionally, input devices may include, but are not limited to, buttons, knobs, switches, keypads, touchscreens, remote controls, etc., and output devices may include, but are not limited to, displays, indicators, gauges, etc. Some embodiments of computer system 400 interface with additional computers, peripheral devices, etc., such that device I / O subsystem 105 can include various physical and / or logical interfaces (e.g., ports, etc.) to facilitate interaction and control between components.

[0069]

[0074] Computer system 400 may further include (and / or communicate with) one or more non-transitory storage devices 425, which may include, but are not limited to, local and / or network-accessible storage, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices such as random access memory (“RAM”), and / or read-only memory (“ROM”), which may be programmable, flash-updateable, etc. Such storage devices may be configured to implement suitable data stores, including, but not limited to, various file systems, database structures, etc. In some embodiments, storage device 425 includes non-transitory memory 240. In some embodiments, storage device 425 may include MS data store 140, HP event store 153, and / or assessment data store 156.

[0070]

[0075] The computer system 400 may also include a communications subsystem 430, which may include, but is not limited to, any suitable antenna, transceiver, modem, network card (wireless or wired), infrared communications device, wireless communications device, chipset (e.g., Bluetooth device, 802.11 device, WiFi device, WiMax device, cellular communications device, etc.), and / or other communications components. As shown, the communications subsystem 430 may also include the network interface 220 for facilitating communications between the user device and a remote server over a communications network. The communications subsystem 430 may further facilitate communications with other computing systems.

[0071]

[0076] In many embodiments, computer system 400 further includes working memory 435, which may include RAM or ROM devices, as described herein. Computer system 400 may also include software elements shown as currently residing in working memory 435, including an operating system 440, device drivers, executable libraries, and / or other code, such as one or more application programs 445, which may include computer programs provided by various embodiments and / or may be designed to perform methods and / or configure systems provided by other embodiments, as described herein. By way of example only, one or more procedures described with respect to the methods described herein may be implemented as code and / or instructions executable by a computer (and / or a processor within a computer), and in some aspects, such code and / or instructions may be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations in accordance with the described methods. In some embodiments, operating system 440 and working memory 435 are used in conjunction with one or more processors 410 to implement the functionality of rhythm estimator subsystem 150.

[0072]

[0077] These sets of instructions and / or code may be stored on a non-transitory computer-readable storage medium, such as non-transitory storage device 425 described above. In some cases, the storage medium may be incorporated within a computer system, such as computer system 400. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium such as a compact disc) and / or provided in an installation package, where the storage medium may be used to program, configure, and / or adapt a general-purpose computer having the instructions / code stored thereon. These instructions may be in the form of executable code that can be executed by computer system 400 and / or may be in the form of source and / or installable code when compiled and / or installed on computer system 400 (e.g., using various commonly available compilers, installation programs, compression / decompression utilities, etc.) and then take the form of executable code.

[0073]

[0078] Those skilled in the art will recognize that substantial modifications can be made in accordance with particular requirements. For example, customized hardware could be used and / or particular elements could be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connection to other computing devices, such as network input / output devices, may be employed.

[0074]

[0079] As noted above, in one aspect, some embodiments may employ a computer system (such as computer system 400) to perform methods according to various embodiments of the present invention. According to one set of embodiments, some or all of the steps of such methods are performed by computer system 400 in response to processor 410 executing one or more sequences of one or more instructions (which may be embedded in other code, such as operating system 440 and / or application program 445) contained in working memory 435. Such instructions may be loaded into working memory 435 from another computer-readable medium, such as one or more non-transitory storage devices 425. By way of example only, execution of the sequences of instructions contained in working memory 435 may cause processor 410 to perform one or more steps of the methods described herein.

[0075]

[0080] As used herein, the terms “machine-readable medium,” “computer-readable storage medium,” and “computer-readable medium” refer to any medium that participates in providing data that causes a machine to operate in a specific manner. These media may be non-transitory. In embodiments implemented using computer system 400, various computer-readable media may be involved in providing instructions / code to processor 410 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical and / or magnetic disks, such as non-transitory storage device 425. Volatile media include, but are not limited to, dynamic memory, such as working memory 435. Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or other magnetic media, CD-ROMs, other optical media, other physical media having patterns of marks, RAM, PROM, EPROM, FLASH-EPROM, other memory chips or cartridges, or other media from which a computer can read instructions and / or code. Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 410 for execution. By way of example only, the instructions may initially be carried on a magnetic and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium to be received and / or executed by the computer system 400. The communications subsystem 430 (and / or its components) typically receives the signals, and the bus 405 can then carry the signals (and / or the data, instructions, etc. carried by the signals) to the working memory 435, from which the processor 410 retrieves and executes the instructions.The instructions received by the working memory 435 may optionally be stored on a non-transitory storage device 425 either before or after execution by the processor 410 .

[0076]

[0081] Furthermore, it should be understood that components of computer system 400 can be distributed across a network. For example, some processing may be performed in one location using a first processor, while other processing may be performed by another processor remote from the first processor. Other components of computer system 400 may be similarly distributed. Thus, computer system 400 may be interpreted as a distributed computing system that performs processing at multiple locations. In some cases, computer system 400 may be interpreted as a single computing device, such as separate laptops, desktop computers, etc., depending on the context.

[0077]

[0082] FIG. 5 shows a flow diagram of an exemplary method 500 for haptic-based assessment of rhythmic performance with reference to a musical score, according to various embodiments. Method 500 may be implemented using any suitable system, including those described above in FIGS. 1-4. An embodiment of method 500 begins at stage 504 by outputting, via an output device interface of a computing device, an audiovisual representation of real-time progression through a musical score at a tempo. The musical score includes a musical notation representation of each of a sequence of SNR events. Each SNR event has a stored logical definition that includes an associated SNR onset time and an associated SNR offset time. In some embodiments, some of the SNR events are note events, and others of the SNR events are rest events. The musical notation representation of each note event indicates one or more notes of one or more pitches to be held for a notated duration beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event. The music notation representation of each rest event indicates silence to be maintained for a notated duration beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event.

[0078]

[0083] In some embodiments, the outputting step at stage 504 includes displaying the musical score via an output device interface, playing audio corresponding to at least each of the sequence of SNR events to audibly represent real-time progression through the musical score at a tempo, and displaying, during playback, a graphical position indicator (e.g., a cursor, a box around the currently playing SNR event, etc.) overlaid on the musical score that dynamically updates to represent which of the sequence of SNR events corresponds to the currently played audio, such that the graphical position indicator visually represents the real-time progression of the musical score at the tempo. In some such embodiments, the outputting step at stage 504 further includes displaying, during playback, a graphical representation of an associated HP onset time and an associated HP offset time of each of the plurality of HP events as each is acquired on the musical score. For example, such an overlay provides a user with real-time feedback indicating how haptic interactions are being received (e.g., including when the HP onset time and HP offset time are being recorded relative to the score playback). In some such embodiments, the outputting step at stage 504 further includes playing a rhythm track (eg, a metronome, a backing track, etc.) in tempo synchronized with the playback of the audio.

[0079]

[0084] At stage 508, embodiments may obtain HP events corresponding to user interactions with a haptic device interface of a computing device during the outputting step. The HP events represent a rhythmic performance by a user of a musical score at a tempo. Each HP event has an associated HP onset time and an associated HP offset time.

[0080]

[0085] At stage 512, embodiments may generate evaluation data for the SNR events by a rhythm evaluator subsystem in communication with the output device interface and the haptic device interface. The generating step at stage 512 may include performing stages 516 and 520. At stage 516, embodiments may obtain, for each of the SNR events, an associated evaluation metric for the SNR event based on an associated SNR onset time and an associated SNR offset time. At stage 520, embodiments may compare the associated evaluation metric for each SNR event with an associated HP onset time and an associated HP offset time for each of at least some of the HP events. The comparing step at stage 520 may include identifying matching events as those of the HP events, each determined to correspond in time to one of the SNR events, and evaluating a rhythmic match between each of the plurality of matching events and a temporally corresponding one of the plurality of SNR events.

[0081]

[0086] In some embodiments, the outputting step at stage 508 establishes a performance time metric such that each HP onset time and each HP offset time corresponds to a respective time value of the performance time metric. In such embodiments, generating evaluation data at stage 512 can include mapping, for each SNR event, an associated SNR onset time and an associated SNR offset time to the performance time metric. The evaluation metric at stage 516 can be defined relative to the performance time metric, and the comparing step at stage 520 can be performed relative to the performance time metric.

[0082]

[0087] In some embodiments, obtaining the relevant evaluation metrics in stage 516 includes calculating a valid match window for each SNR onset time. Some such embodiments may set the duration of the valid match window based on tempo and / or based on determining a performance skill level based on a predetermined skill level of the user and / or a predetermined difficulty level of the musical score. In some such embodiments, the comparison at stage 520 may identify a matching event as one of the HP events each determined to correspond in time to one of the multiple SNR events by: searching for the HP event to determine whether any of the multiple HP events has an associated HP onset time that is within the valid match window for the SNR event; identifying the SNR event as a skipped SNR event in response to determining that none of the multiple HP events has an associated HP onset time that is within the valid match window of the SNR event; identifying the SNR event as an executed SNR event in response to determining that a particular HP event of the multiple HP events has an associated HP onset time that is within the valid match window of the SNR event, such that the executed SNR event has an executed onset time that corresponds to the HP onset time of the particular HP event and an executed offset time that corresponds to the HP offset time of the particular HP event, and associating the SNR event with the particular HP event as a matching event of the multiple matching events determined to correspond in time to the SNR event.

[0083]

[0088] At stage 524, embodiments may display visual feedback via an output device interface to graphically map the evaluation data of the plurality of SNR events to a music notation representation of the plurality of SNR events on the musical score. In some embodiments, the displaying at stage 524 includes, for each executed SNR event, graphically mapping an executed onset time and an executed offset time of the executed SNR event to a music notation representation of the SNR event on the musical score. In some embodiments, the displaying at stage 524 includes, for each executed SNR event, first graphically mapping the executed onset time and executed offset time of the executed SNR event to a music notation representation of the SNR event on the musical score, and second graphically mapping the associated SNR onset time and associated SNR offset time of the executed SNR event to a music notation representation of the executed SNR event on the musical score, wherein the first graphical mapping step is visually distinguishable from the second graphical mapping step. In some embodiments, the displaying at stage 524 includes, for each skipped SNR event, graphically mapping the SNR event's associated SNR onset time and associated SNR offset time to a music notation representation of the skipped SNR event on the musical score. In some embodiments, obtaining the associated evaluation metric for each SNR event at stage 516 includes calculating a valid offset window for the associated SNR offset time, and the displaying at stage 524 includes, for each performed SNR event, graphically mapping the complete performance instruction to a music notation representation of the performed SNR event on the musical score only in response to determining that the performed offset is within the valid offset window.

[0084]

[0089] The methods, systems, and devices described above are examples. In various configurations, various procedures and components may be omitted, substituted, or added, as appropriate. For example, in alternative configurations, methods may be performed in an order different from that described, and / or various stages may be added, omitted, and / or combined. Also, features described with respect to particular configurations may be combined in various other configurations. Different aspects and elements of the configurations may be combined in a similar manner. Also, because technology evolves, many of the elements are examples and do not limit the scope of the disclosure or the claims.

[0085]

[0090] Specific details are set forth in the description to provide a thorough understanding of example configurations (including implementations). However, configurations may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the configurations. This description provides example configurations only and is not intended to limit the scope, applicability, or configuration of the claims. Rather, the foregoing description of the configurations will provide one of ordinary skill in the art with an enabling description for implementing the described technology. Various changes can be made in the function and arrangement of elements without departing from the spirit or scope of the present disclosure.

[0086]

[0091] Configurations may also be described as processes that are depicted as flow diagrams or block diagrams. While each operation may be described as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may have additional steps not included in the diagrams. Furthermore, example methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. If implemented by software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a non-transitory computer-readable medium, such as a storage medium. A processor may perform the described tasks.

[0087]

[0092] While several example configurations have been described, various modifications, alternative configurations, and equivalents may be used without departing from the spirit of this disclosure. For example, the above elements may be components of a larger system, other rules may take precedence over or otherwise modify the application of the present invention, and certain steps may be performed before, during, or after considering the above elements.

Claims

1. 1. A method for haptic-based assessment of a rhythmic performance with reference to a musical score, said method comprising: outputting, via an output device interface of a computing device, an audiovisual representation of real-time progression through a musical score at a tempo, said musical score including a musical notation representation of each of a sequence of scored-notated rhythmic (SNR) events, each SNR event having a stored logical definition including an associated SNR onset time and an associated SNR offset time; during the outputting step, acquiring a plurality of haptic performance (HP) events corresponding to user interactions with a haptic device interface of the computing device representing a rhythmic performance by the user of the musical score at the tempo, each HP event having an associated HP onset time and an associated HP offset time; Steps below: For each SNR event of the plurality of SNR events, obtaining an associated metric for the SNR event based on the associated SNR onset time and the associated SNR offset time; and comparing the associated evaluation metric of each SNR event with the associated HP onset time and the associated HP offset time of each of at least some HP events of the plurality of HP events, wherein the comparison identifies a plurality of matching events as those HP events each determined to correspond in time to one of the plurality of SNR events, and evaluating rhythmic compatibility between each of the plurality of matching events and the corresponding one of the plurality of SNR events; generating evaluation data for the plurality of SNR events by a rhythm estimator subsystem in communication with the output device interface and the haptic device interface by displaying, via the output device interface, visual feedback for graphically mapping the rating data of the plurality of SNR events to the music notation representation of the plurality of SNR events on the musical score; A method comprising:

2. the output device of the computing system is a touchscreen interface; wherein the step of outputting the audiovisual representation includes outputting a graphical performance interface via the touchscreen interface, whereby: a first portion of the graphical performance interface displaying the musical score having the musical notation representation of the plurality of SNR events; a second portion of the graphics performance interface designated for tactile input, whereby the step of obtaining the plurality of HP events corresponds to the user interaction with the haptic device interface only within the second portion of the graphics performance interface; The method of claim 1.

3. said step of outputting said audiovisual representation further comprising: outputting a graphical representation of a boundary of the second portion of the graphical performance interface at which the user interaction with the haptic device interface is recognized as a Home Page event, the second portion of the graphical performance interface not completely overlapping the first portion of the graphical performance interface; The method of claim 2 further comprising:

4. a second portion of the graphics performance interface, a first area graphically designated by the haptic device interface for left-hand user interaction, the left-hand user interaction being recognized as a left-hand HP event; a second area graphically designated by the haptic device interface for right-hand user interaction, the right-hand user interaction being recognized as a right-hand HP event, the second area not completely overlapping the first area; and The method of claim 2 , comprising:

5. establishing a performance time base such that the outputting step each HP onset time and each HP offset time corresponds to a respective time value of the performance time base; generating the evaluation data for the plurality of SNR events includes, for each SNR event, mapping the associated SNR onset time and the associated SNR offset time to the performance time metric, wherein the evaluation metric for each SNR event is defined relative to the performance time metric; and comparing the associated evaluation metric for each SNR event being performed to the performance time metric. The method of claim 1.

6. obtaining the associated metric for each SNR event includes calculating a valid match window for the SNR onset time; The comparing step includes: searching the plurality of HP events to determine whether any of the plurality of HP events has an associated HP onset time that is within the valid match window of the SNR event; in response to determining that none of the plurality of HP events has an associated HP onset time that is within the valid match window of the SNR event, identifying the SNR event as a skipped SNR event; in response to determining that a particular HP event of the plurality of HP events has an associated HP onset time that is within the valid match window of the SNR event, identifying the SNR event as an executed SNR event, such that the executed SNR event has an executed onset time that corresponds to the HP onset time of the particular HP event and an executed offset time that corresponds to the HP offset time of the particular HP event, and associating the SNR event with the particular HP event as the matching event of the plurality of matching events determined to correspond in time to the SNR event; identifying the plurality of matching events as the HP events each determined to correspond in time to one of the plurality of SNR events by The method of claim 1.

7. The comparing step setting the duration of the valid match window based on the tempo The method of claim 6 further comprising:

8. generating the evaluation data includes determining a performance skill level based on a predetermined skill level of the user and / or a predetermined difficulty level of the musical score; the comparing step further comprising setting a duration of the valid match window based on the performance skill level. The method of claim 6.

9. wherein the step of displaying the visual feedback to graphically map the evaluation data comprises, for each executed SNR event, graphically mapping the executed onset time and the executed offset time of the executed SNR event to the music notation representation of the SNR event on the music score. The method of claim 6.

10. wherein the step of displaying the visual feedback to graphically map the evaluation data comprises, for each SNR event performed: first graphically mapping the executed onset time and the executed offset time of the executed SNR event to the music notation representation of the SNR event on the music score; second graphically mapping the associated SNR onset time and the associated SNR offset time of the executed SNR event onto the music notation representation of the executed SNR event on the music score; Including, the first graphically mapping step being visually distinguishable from the second graphically mapping step. The method of claim 6.

11. wherein the step of displaying the visual feedback to graphically map the evaluation data comprises, for each skipped SNR event, graphically mapping the associated SNR onset time and the associated SNR offset time of the SNR event to the music notation representation of the skipped SNR event on the music score. The method of claim 6.

12. the step of obtaining the associated metric for each SNR event further comprises calculating a valid offset window for the associated SNR offset time; wherein the step of displaying the visual feedback includes, for each executed SNR event, graphically mapping a complete performance indication onto the music notation representation of the executed SNR event on the musical score only in response to determining that the executed offset is within the valid offset window. The method of claim 6.

13. wherein the outputting step includes displaying the music notation representation of the sequence of SNR events on a grand staff, such that the sequence of SNR events includes a right-hand SNR event and a left-hand SNR event; the step of acquiring the plurality of HP events includes the steps of acquiring a first portion of the HP events from a designated right-hand portion of the haptic device interface as right-hand HP events, and acquiring a second portion of the HP events from a designated left-hand portion of the haptic device interface as left-hand HP events. The method of claim 1.

14. generating the evaluation data for the plurality of SNR events; comparing the associated metric of each of the right hand SNR events to the associated HP onset time and the associated HP offset time of each of the right hand HP events only; comparing the associated metric for each of the right-hand SNR events to the associated HP onset time and the associated HP offset time for each of only the left-hand HP events; The method of claim 13,

15. said step of outputting said audiovisual representation of real-time progression through said musical score at said tempo comprising: displaying the musical score via the output device interface; playing audio corresponding to at least each of the sequence of SNR events to aurally represent the real-time progression through the musical score at the tempo; during said playback, displaying, overlaid on said musical score, a graphical position indicator that dynamically updates to indicate which of said sequence of SNR events corresponds to said audio currently being played, whereby said graphical position indicator visually represents said real-time progression through said musical score at said tempo; The method of claim 1 , comprising:

16. said step of outputting said audiovisual representation of real-time progression through said musical score at said tempo comprising: during said playback, as each of said plurality of HP events is acquired, displaying a graphical representation of said associated HP onset time and said associated HP offset time of each of said plurality of HP events overlaid on said musical score.

16. The method of claim 15, further comprising:

17. said step of outputting said audiovisual representation of real-time progression through said musical score at said tempo comprising: playing a rhythm track at said tempo in synchronization with the playback of said audio; 16. The method of claim 15, further comprising:

18. each of the first subset of the plurality of SNR events corresponds to a musical note event, and the music notation representation of each musical note event indicates one or more musical notes of one or more pitches to be held for a notated duration beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event; each of the second subset of the plurality of SNR events corresponds to a rest event, and the music notation representation of each rest event indicates silence to be maintained for the notated duration beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event. The method of claim 1.

19. the rhythm estimator subsystem is implemented by one or more processors of the computing device; The method of claim 1.

20. the rhythm estimator subsystem is implemented by a server in communication with the computing device over a communications network; The method of claim 1.

Images