Automatic evaluation of haptic-based rhythmic performances, referencing musical scores.

Abstract

A technique is described for automated evaluation of haptic-based rhythmic performance with reference to a musical score. An audiovisual representation of real-time progression through the musical score is output to a user. The output corresponds to the progression of a sequence of scored rhythmic (SNR) events at a tempo. During the output, haptic performance (HP) events are received from the user via a haptic interface as the user's rhythmic performance of the SNR events at the tempo. The automated technique evaluates the user's performance to determine a rhythmic match between the received HP events and evaluation data of the SNR events. Visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

Description

Technical Field

[0001]

[0001] Embodiments generally relate to performance feedback applications, and more particularly, to the automatic evaluation of rhythmic tactile performance events for notated music score events.

Background Art

[0002]

[0002] The ability to sight-read musical rhythms is a critically important skill in music education at all levels. Standard musical notation includes a large symbol system for specifying rhythmic features. For example, a typical music score indicates musical events over linear time. Musical events can include at least notes (generally, the specification of filled musical space, such as by pitch) and rests (a general specification of empty musical space, such as by silence). Each musical event can be represented on the music score in different forms to indicate the duration of the musical event. For example, a particular note or rest can be specified as consuming a duration such as a whole note, half note, quarter note, eighth note, dotted quarter note, double dotted quarter note, triplet, etc. By mapping the indicated duration to a particular tempo (which may also be specified by the music score, by the musical genre, or by some other means), a musician can know how long to hold a note or rest.

[0003]

[0003] Learning to read and play such rhythmic patterns can generally follow several stages. In one stage, the musician learns the notation and their explicit meanings. For example, a beginner-level musician might learn about quarter notes, the relationship between the duration of a quarter note and the duration of a measure based on one or more time signatures, the relative duration of a quarter note to other elements of a measure or piece, etc. In another stage, the musician might learn to play quarter notes, for example, by practicing the feel of playing quarter notes at one or more tempos. In yet another stage, the musician might learn and practice rhythmic patterns that include quarter notes and other musical elements (e.g., quarter rests, half notes, etc.). In yet another stage, the musician might learn to play rhythmic patterns in the context of a piece of music that includes additional musical features such as pitch and dynamics. More advanced players may encounter difficult and / or unfamiliar rhythmic patterns and may revisit one or more of the same stages.

[0004]

[0004] Many conventional techniques have been used by music students for centuries to learn rhythmic elements and patterns. More recently, several computer-based music training applications have incorporated similar techniques that allow students to learn music without a live teacher. Some such applications train users to play songs of music in which rhythm is learned only in the context of a particular song. For example, a recording of a song is played for the user as a small part of a musical score (e.g., one or two measures of a musical score), and the user attempts to copy what they hear, including pitch and rhythm. While such applications can help students learn a particular song, students may struggle to translate or extend that knowledge to new songs and / or new rhythms, and students may find that each new song is a new exercise in imitation. Furthermore, users of such applications may learn to play a large number of songs by encountering only relatively small variations in rhythm. Other such applications teach decontextualized rhythm from songs of any music. For example, a recorded rhythm pattern may play itself, synchronized with flashing graphics, and the user may be prompted to repeat the rhythm pattern through the interface. While such applications can help students improve their rhythmic recall and efficiently provide users with a wide variety of rhythm patterns, students using such applications may never know how such rhythms are represented in the context of a musical piece. These and other conventional music training applications tend to be limited in their ability to provide students (especially more advanced students) with effective rhythm training contextualized to standard sheet music notation of musical pieces. [Overview of the Initiative] [Means for solving the problem]

[0005]

[0005] Embodiments of the present invention relate to automated evaluation of haptic-based rhythmic performance referencing a musical score. An audiovisual representation of the progress in real time through the musical score is output to the user. The output corresponds to the progress of a sequence of score-noted rhythm (SNR) events at a given tempo. During the output, haptic performance (HP) events are received from the user via a haptic interface as the user's rhythmic performance of the SNR events at that tempo. Automated technology evaluates the user's performance and determines the rhythmic fit between the evaluated data of the received HP events and SNR events. Visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

[0006]

[0006] According to a series of embodiments, a method for haptic-based evaluation of rhythmic performance with reference to a musical score is provided. The method comprises the steps of: outputting an audiovisual representation of real-time progress through a musical score at tempo via the output device interface of a computing device, wherein the musical score includes a musical notation representation of each of a sequence of score-noted rhythm (SNR) events, and each SNR event has a stored logical definition including an associated SNR onset time and an associated SNR offset time; acquiring a plurality of haptic performance (HP) events corresponding to user interaction with the computing device's haptic device interface representing the user's rhythmic performance of the musical score at tempo during the output step, wherein each HP event has an associated HP onset time and an associated HP offset time; and for each SNR event of the plurality of SNR events, acquiring an associated evaluation criterion for the SNR event based on the associated SNR onset time and associated SNR offset time. The step of comparing the relevant evaluation criteria for each SNR event with the relevant HP onset time and relevant HP offset time of each of at least some of the HP events among a plurality of HP events, wherein the comparison step includes: identifying the plurality of matching events as belonging to the HP events which are determined to be temporally corresponding to one of the plurality of SNR events; and evaluating the rhythmic compatibility between each of the plurality of matching events and the temporally corresponding one among the plurality of SNR events; the step of generating evaluation data for the plurality of SNR events by a rhythm evaluator subsystem that communicates with an output device interface and a haptic device interface; and displaying visual feedback via the output device interface for graphically mapping the evaluation data for the plurality of SNR events to a musical notation representation of the plurality of SNR events on a musical score.

[0007]

[0007] This summary is not intended to identify any important or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate parts of the entire specification of this patent, any or all of the drawings, and each of the claims.

[0008]

[0008] The above, along with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0009]

[0009] This disclosure will be explained in conjunction with the attached drawings. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an exemplary tactile rhythm training system according to various embodiments described herein. [Figure 2] This figure shows another exemplary haptic rhythm training system implemented in a cloud-based environment according to various embodiments described herein. [Figure 3A] This figure shows exemplary screenshots from exemplary applications for implementing the features of a tactile rhythm training system according to various embodiments described herein. [Figure 3B] Another figure shows an exemplary screenshot from an exemplary application for implementing the features of a haptic rhythm training system according to various embodiments described herein. [Figure 3C] Another figure showing exemplary screenshots from exemplary applications for implementing the features of a tactile rhythm training system according to various embodiments described herein. [Figure 4] This is a schematic diagram of one embodiment of a computer system that can implement various system components and / or perform various steps of a method provided by various embodiments. [Figure 5] This is a flowchart illustrating exemplary methods for tactile-based evaluation of rhythmic performances referencing musical scores, using various embodiments. [Figure 6A] This figure shows exemplary screenshots from exemplary applications for implementing a history user feedback function of a haptic rhythm training system according to various embodiments described herein. [Figure 6B] Another figure shows an exemplary screenshot from an exemplary application for implementing a history user feedback function of a haptic rhythm training system according to various embodiments described herein. [Modes for carrying out the invention]

[0011]

[0016] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a second label (e.g., lowercase) that distinguishes similar components. Where only the first reference label is used herein, the description is applicable to any similar component having the same first reference label, regardless of the second reference label.

[0012]

[0017] Embodiments of the disclosed technology will become clearer when considered in relation to the description of the drawings below. The following description includes numerous specific details to provide a complete understanding of the invention. However, those skilled in the art should recognize that the invention may be carried out without these specific details. In some cases, circuits, structures, and techniques are not shown in detail to avoid obscuring the invention.

[0013]

[0018] The ability to sight-read musical rhythms is a crucial skill in music education at all levels. A typical musical score contains the notation of musical events over linear time. Sight-reading music involves the real-time performance of these musical events. For example, a typical musical score may include one or more staves that place at least notes (generally designations of filled musical space, such as by pitch) and rests (general designations of empty musical space, such as by silence). For most instruments, the vertical dimension of the staff relates to pitch, and the horizontal dimension of the staff relates to time. A musical score may additionally include designations of time signatures, key signatures, dynamics, and / or many other musical features.

[0014]

[0019] As used herein, notes and rests are referred to as notated rhythmic (SNR) events, as they are represented in musical scores. Each SNR event may include associated SNR onset time and SNR offset time. As used herein, “onset time” is the time when the rhythmic event begins, and “offset time” is the time when the rhythmic event ends. The relative horizontal arrangement of SNR events on the staff can indicate the SNR onset time, and standard notation symbols specify the duration of the SNR event, thereby indicating the SNR offset time. Standard musical notation includes a large symbolic system for specifying SNR events. For example, various symbols are used to indicate that a note or rest consumes a duration such as a whole note, half note, quarter note, eighth note, dotted quarter note, double-dotted quarter note, triplet eighth note, or quarter note coupled with a sixteenth note. The played duration of an SNR event can be determined by a combination of the notated duration (e.g., quarter note) and the tempo (e.g., beats per minute). In some cases, the duration of a performance is determined by additional and / or alternative factors such as the musical genre, time signature, and additional score markings (e.g., indicating to play certain passages more quickly, or to gradually slow down a passage). By recognizing and understanding the entire rhythmic symbol system, a musician can know when to play each SNR event and how long to hold each SNR event, thereby translating the written music into the played music.

[0015]

[0020] The ability to translate notated SNR events into performed SNR events, particularly in real time, can involve extensive training and practice. Even skilled musicians may struggle to play complex and / or unfamiliar rhythmic patterns. Embodiments of this specification provide a novel technique for providing rhythm training to musicians by providing a real-time score-referenced association between SNR events provided by the application and haptic performance (HP) events provided by the user. An audiovisual representation of the real-time progression (including the progression through a sequence of SNR events at tempo) through a musical score is output to the user. In some implementations, the tempo is user-controllable. For example, the user can speed up or slow down the tempo using a slider bar or any other appropriate interface control. HP events, on the other hand, are received from the user in response to user interaction with a haptic device interface representing the rhythmic performance of the SNR events at tempo by the user. The user's performance is evaluated to determine the rhythmic fit between the received HP events and evaluation data of the SNR events, and visual feedback is provided to the user to graphically map the evaluated performance to the SNR events on the musical score.

[0016]

[0021] Referring first to Figure 1, exemplary haptic rhythm training systems 100 according to various embodiments described herein are shown. System 100 includes a device input / output (I / O) subsystem 105, a rhythm evaluator subsystem 150, and a music score (MS) data store 140. Embodiments of the device I / O subsystem 105 include a haptic interface 110 having a haptic event processor 115, a display interface 120 having a display processor 125, and an audio interface 130 having an audio processor 135. Embodiments of system 100 are implemented on a computing device. In some embodiments, the computing device is a portable electronic device such as a laptop computer, tablet computer, or smartphone. In other embodiments, the computing device is a machine such as a desktop computer. In other embodiments, the computing device is integrated with a musical instrument such as an electronic piano.

[0017]

[0022] Embodiments of the haptic rhythm training system 100 provide a rhythm training environment based on detecting and evaluating the rhythm correspondence between the user's rhythmic performance (based on detected haptic performance (HP) events) and an audiovisual representation of the real-time progression through a sequence of score-noted rhythm (SNR) events notated on a musical score. Details of the musical score, including the sequence of SNR events, are stored in the MS data store 140. As illustrated, the MS data store 140 stores at least MS visual data 145 and MS logical data 143. In some embodiments, the MS visual data 145 includes all information used by the display processor 125 to generate a graphic score representation, including a graphic representation of the SNR events, for visual output by the display interface 120. For example, the MS visual data 145 includes a stored graphic representation of the score, including notes and rests, and any additional graphic score elements (e.g., staves, bar lines, note symbols, accidentals, time signatures, key signatures, dynamics, tempo markings, expression markings, titles, lyrics, etc.). In other embodiments, the MS visual data 145 includes only a portion of the information used by the display processor 125 to generate a visual output of the musical score via the display interface 120. For example, the MS visual data 145 includes score elements other than notes and rests, and the display processor 125 generates a visual representation of notes and rests based on the MS logical data 143 for SNR events.

[0018]

[0023] The MS logical data 143 logically defines at least a sequence of SNR events, indicating at least the associated SNR onset time and SNR offset time for each SNR event. In some implementations, the MS logical data 143 for each SNR event includes a type identifier indicating whether the SNR event represents a filled music space (e.g., a note) or an empty music space (e.g., a rest). In some implementations, the MS logical data 143 includes other information about a particular SNR event, such as the pitch of a note, whether the SNR event is part of a triplet or other grouping, and whether the SNR event is associated with a particular hand or finger.

[0019]

[0024] In some embodiments, MS logic data 143 is score-referenced so that the SNR onset time and / or SNR offset time (and / or SNR duration) are defined in relation to the notation of the underlying musical score. The relationship to the score may be based on the type of note or rest (e.g., "quarter note"), the consumed portion of one or more measures (e.g., half a measure), the number of metronome beats (e.g., 3 beats), etc. In an exemplary SNR event, the SNR onset time is shown as occurring on the second beat of the fourth measure, and the SNR offset time is shown as occurring on the fourth beat of the fourth measure. If necessary, additional MS logic data 143 may be used to convert the SNR offset time to a duration. For example, MS logic data 143 may indicate that the musical score is within a "4 / 4" time, from which it may be derived that the example SNR event is a half note. Additionally or alternatively, the SNR offset time may be shown as a duration within the MS logic data 143 of the score reference. Referring to the same exemplary SNR event, the MS logical data 143 may indicate that the SNR event has a duration of 2 beats, or that the SNR event is a half note. In such cases, the SNR offset timing can be derived, if necessary, by adding the explicitly indicated duration to the SNR onset time. In either case, the tempo can be used to convert the score-referenced MS logical data 143 into real-time reference timing information. For example, the additional MS logical data 143 indicates a default tempo of 120 beats / minute, and the score-referenced half note consumes approximately 1 second of real-time at the default tempo. The score reference definition of the MS logical data 143 provides various features. For example, the real-time duration of all SNR events can be automatically adjusted along with tempo changes.

[0020]

[0025] Alternatively, the MS logical data 143 can be referenced in real time. In one implementation, each SNR event is defined as a start (i.e., the SNR onset time) of a number of milliseconds from the start of the song and an end (i.e., the SNR offset time) of a number of milliseconds from the start of the song. In another implementation, each SNR event is defined as a start (i.e., the SNR onset time) of a number of milliseconds from the start of the song and an end (i.e., the SNR offset time) of a number of milliseconds after the SNR onset time. Regardless of whether the MS logical data 143 is score-referenced or referenced in real time, the SNR offset time is defined as the associated end time of the SNR event or as the duration following the associated SNR onset time of the SNR event.

[0021]

[0026] As described herein, embodiments of the rhythm evaluator subsystem 150 can evaluate a user's rhythmic performance only in relation to limited SNR event information. In some embodiments, the rhythm evaluator subsystem 150 evaluates only with respect to SNR onset time and SNR offset time. In other embodiments, the rhythm evaluator subsystem 150 evaluates further based on whether the SNR event is a note or a rest. In other embodiments, the rhythm evaluator subsystem 150 evaluates further based on whether the SNR event is associated with a particular hand, finger, etc. For example, in these embodiments, the rhythm evaluator subsystem 150 can evaluate user performance without considering pitch, dynamics, key signature, etc. Thus, embodiments of the MS logical data 143 include only information useful for rhythm evaluation by the rhythm evaluator subsystem 150. In some such embodiments, the display processor 125 generates a graphic representation of the musical score and SNR events based solely on the MS visual data 145 without processing the MS logical data 143 in any way. In other such embodiments, the display processor 125 partially generates a graphic representation of an SNR event based on MS logical data 143 (e.g., determining the horizontal placement of notes and rests corresponding to an SNR event based on the SNR onset time, determining the note or rest type based on the SNR offset time, etc.), while the remaining graphic representation of the musical score and / or SNR event elements is generated from MS visual data 145. For example, MS logical data 143 is used to determine timing-related information of the SNR event, and MS visual data 145 is used to determine the pitch. Some embodiments of the display processor 125 include a notation rule engine (not shown) that can derive additional information from MS visual data 145 and / or MS logical data 143, which can be used to generate a graphic representation of the musical score and / or SNR event elements.For example, a notation rule engine can determine whether a note has a flag, whether notes are linked to each other across a measure boundary, whether the note stem is pointing upwards or downwards, and whether an accidental is indicated.

[0022]

[0027] Embodiments of the display interface 120 and the display processor 125 are implemented in any suitable manner to support the visual portion of the audiovisual output function described herein. The terms “visual” and “graphical” are used interchangeably herein. The display interface 120 can be implemented by any suitable display component, such as a tablet computer display, a smartphone display, or a computer monitor. The display processor 125 can be implemented as any processor, part of a processor, or group of processors capable of generating graphic output via the display interface 120, as described herein. For example, the display processor 125 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), etc. In some implementations, a touchscreen interface integrates the display interface 120 and the haptic interface 110. As described above, the display processor 125 can use MS visual data 145 (and, for example, MS logical data 143) to generate a graphic representation of a musical score, including a graphic representation of a sequence of SNR events to be performed by the user.

[0023]

[0028] Embodiments of the display processor 125 provide additional features. Some such features include generating a display output for graphically representing real-time progress through a musical score, such as graphically indicating the current playback position. Other such features include generating a display output for graphically representing evaluation feedback. These and other features of the display interface 120 and the display processor 125 are described further below.

[0024]

[0029] Embodiments of the audio interface 130 and the audio processor 135 are implemented in any suitable manner to support the auditory portion of the audiovisual output functionality described herein. The terms “auditory” and “audio” are used interchangeably herein. The audio interface 130 may be implemented by any suitable audio component, for example, one or more audio transducers, speakers, headphones, etc. The audio processor 135 may be implemented as any processor, part of a processor, or group of processors capable of generating audio output through the audio interface 130 as described herein. In some embodiments, the audio representation of an SNR event is dynamically generated from MS logic data 143. In some such embodiments, the audio representation of a sequence of SNR events indicates only the SNR onset time by a buzzer, vibration, or click, etc., at each SNR onset time. In other such embodiments, the audio representation further indicates the SNR offset time of the SNR event (for example, the audio interface 130 buzzes or vibrates over the duration between the SNR onset and SNR offset of each SNR event). In other such embodiments, where the MS logical data 143 includes additional audio information, the audio representation of the SNR event indicates further audio information such as pitch and dynamics. In other embodiments, the audio representation of the SNR event is generated from recorded audio. For example, a sequence of SNR events is reproduced by playing a digital audio file from an MS datastore 140 that stores a recording of a musical piece.

[0025]

[0030] Embodiments of the audio processor 135 provide additional features. Some such features include generating additional audio during the user's performance, such as outputting a metronome and / or backing track to indicate tempo. Other such features include generating an audio output for playing back portions of the user's performance for comparison with playback of, for example, an ideal version of the song, a recorded version of the song, or a past performance by the user. These and other features of the audio interface 130 and the audio processor 135 are described further below.

[0026]

[0031] As described above, the embodiment of system 100 evaluates the user's rhythmic performance by obtaining haptic feedback from the user via the haptic interface 110 while an audiovisual representation of a musical score is presented to the user via the display interface 120 and the audio interface 130. The audiovisual representation shows real-time progress through the musical score at a specific tempo by sequentially highlighting each of the sequences of SNR events at the appropriate timing based on the indicated rhythm and playback tempo. The haptic feedback corresponds to the user's interaction with the haptic interface 110, which represents the user's rhythmic performance of the sequence of SNR events in the musical score at the playback tempo. The haptic feedback is recorded as a sequence of haptic performance (HP) events.

[0027]

[0032] The haptic interface 110 can be implemented using any sensor capable of receiving HP events from the user, such that each HP event includes both the HP onset time and the HP offset time. In some implementations, the haptic interface 110 is implemented by the touch sensor functionality of the computing device's touchscreen interface. For example, when a user touches the touchscreen, the haptic event processor 115 detects and records the touch event as a haptic event. In other implementations, the haptic interface 110 is implemented by an input device integrated with the computing device. For example, the user may touch a trackpad, grip a force sensor, click an integrated mouse button, press an integrated keyboard key, or press a key on an electronic piano keyboard, and the haptic event processor 115 detects and records the user interaction as a haptic event. In other implementations, the haptic interface 110 is implemented by a peripheral input device that communicates with other parts of the computing device (e.g., via a wired or wireless connection). For example, the user may click a button on a nearby mouse or keyboard, grasp a nearby haptic glove, or press a key on an external electronic piano keyboard, and the haptic event processor 115 detects and records the user interaction as a haptic event. The embodiments described herein generally describe a haptic interface 110 that interfaces with the user's hand or fingers, but other types of haptic interfaces 110 may also be used. For example, the haptic interface 110 may interface with the user's foot, the user's breath, etc.

[0028]

[0033] In some implementations, information received from the haptic interface 110 can be supplemented with additional inputs received via other inputs. For example, in addition to (e.g., simultaneously with) interaction with the haptic interface 110, the user can also clap, snap, whistle, sing, tap their feet, etc. Such additional interactions can be detected and processed by audio and / or video inputs (e.g., microphone, camera, etc.). In some cases, such additional interactions can be used to set the initial tempo of a performance, receive additional commands during the user's performance, or make it easier to verify haptic feedback (e.g., to improve reliability decisions).

[0029]

[0034] In any of the above or other implementations, the haptic interface 110 is implemented to allow the user to control the HP onset time and HP offset time. In some such implementations, the user initiates engagement with the haptic interface 110 to establish the HP onset time of an HP event, maintains engagement with the haptic interface 110 for a certain duration, and releases engagement from the haptic interface 110 to establish the HP offset time. For example, the user places a finger on the touchscreen interface to define the HP onset time (e.g., presses a key or button, etc.) and removes their finger from the touchscreen interface to define the HP offset time (e.g., releases a key or button, etc.). Other implementations may provide alternative methods for indicating the HP offset time. For example, after engaging with the haptic interface 110 to establish the HP onset time of an HP event, the user can establish the HP offset time by engaging with the haptic interface 110 again (i.e., a first touch or click indicates the start of an HP event, and a second touch or click indicates the end of an HP event), or by engaging with a different part of the haptic interface 110 or a different haptic interface 110. In some embodiments, the HP onset time and HP offset time are defined in opposite ways. For example, the haptic interface 110 and / or the haptic event processor 115 may be configured such that disengaging from the haptic interface 110 indicates the HP onset time, and engaging with the haptic interface 110 indicates the corresponding HP offset time.

[0030]

[0035] In some embodiments, the haptic interface 110 and the haptic event processor 115 are configured to receive multiple simultaneous tracks of HP events. As an example context, a piano student might want to learn a piece of music that has written right-hand and left-hand parts that should be played simultaneously at different rhythms. In another exemplary context, a piano student might want to practice a rhythmically complex passage in a piece where different fingers are playing at different rhythms (for example, some fingers hold a particular note while other fingers of the same hand change to different notes). In some implementations, a single haptic interface 110 can receive multiple tracks of HP events simultaneously. For example, a multi-touch touchscreen interface, a multi-key keyboard, or a multi-button mouse can receive multiple simultaneous haptic interactions. In other implementations, multiple haptic interfaces 110 are used simultaneously to receive multiple simultaneous haptic interactions.

[0031]

[0036] In some embodiments, additional tactile information can be detected, and / or tactile feedback can be used for additional purposes. As described above, some embodiments of the rhythm evaluator subsystem 150 evaluate user performance based solely on onset and offset timing, without considering pitch, dynamics, key signature, etc. In other embodiments, additional tactile interactions can be detected. For example, a touchscreen or other tactile interface 110 may detect how hard the user is pressing, or a tactile glove, 3D mouse or controller, or other tactile interface 110 may detect velocity, acceleration, orientation, 3D gestures, etc. In some implementations, such additional tactile interactions can be used by the rhythm evaluator subsystem 150 to evaluate dynamics, and / or other characteristics of user performance. In some embodiments, the HP events described above and / or such additional tactile interactions can be used for purposes other than rhythm evaluation, such as user interface navigation, playback control, etc. For example, the user may set the performance tempo by tapping the haptic interface 110 at a specific speed before the performance begins, or the user may calibrate the haptic interface 110 to finger positions, sensitivity levels, etc., by tapping each finger on the haptic interface 110 before the performance.

[0032]

[0037] User interaction with the haptic interface 110 (e.g., one interaction at a time or multiple interactions simultaneously) is detected by the haptic event processor 115, which generates HP event data 117. In some implementations, the HP event data 117 represents a sequence of HP events, each with an associated HP onset time and an associated HP offset time. In other implementations, the HP event data 117 represents a sequence of HP onset times and HP offset times from which a sequence of HP events can be derived. For example, if only a single track of HP events is received, each sequential pair of HP onset times and HP offset times can be treated as defining the start and end of the associated HP event, respectively.

[0033]

[0038] Embodiments of the haptic event processor 115 can use various approaches to recognize multiple simultaneous tracks of HP events. In some implementations where multiple haptic interfaces 110 are used, user interactions with each haptic interface 110 are labeled based on its original haptic interface 110 (for example, based on the logical port from which the haptic data is received). In other implementations where a single haptic interface 110 is used, the haptic interface 110 provides additional data that can be used by the haptic event processor 115 to isolate tracks. In some such implementations where the haptic data comes from a multi-key or multi-button haptic interface 110, the haptic event processor 115 can group user interactions by keys or buttons to isolate simultaneous HP events. For example, on a typing keyboard, the user presses the "S" key at time T0, presses the "K" key at time T1, releases the "K" key at time T2, and releases the "S" key at time T3. If only a single sequence is maintained (HP onset at T0, HP onset at T1, HP offset at T2, HP offset at T3), there is no way to know at T2 whether the HP offset is associated with the first HP onset or the second HP onset. By also recording the associated key, the haptic event processor 115 can group the HP onset at T0 together with the HP offset at T3 as both containing the "S" key, and the haptic event processor 115 can group the HP onset at T1 together with the HP offset at T2 as both containing the "K" key. In other such implementations where the haptic data comes from a multitouch interface (e.g., a multitouch touchscreen or trackpad), the haptic event processor 115 can group the HP onset time and HP offset time to isolate simultaneous HP events based on the location or region of the multitouch interface where each user interaction occurs.As an example, a user may use both hands and / or multiple fingers simultaneously to interact with the touchscreen, thereby inherently generating multiple simultaneous interactions with different locations on the touchscreen, and the haptic event processor 115 can group these interactions by location to determine which HP onset times and which HP offset times to pair. As another example, separate areas of the touchscreen may be pre-designated (e.g., labeled, color-coded, etc.) to accept their respective tracks of HP events (e.g., right-hand area and left-hand area), and the haptic event processor 115 may be configured to process interactions from each area in relation to its pre-designated track.

[0034]

[0039] In some embodiments, the HP onset time always indicates the start of user interaction with the haptic interface 110, and the HP offset time always indicates the end of user interaction with the haptic interface 110. In some such embodiments, a “note” event (whether an SNR event or an HP event) is identified as an onset time followed by a corresponding offset time, and a “rest” event (whether an SNR event or an HP event) is identified as an offset time followed by a corresponding onset time. In other embodiments, a note HP event is characterized by an onset time followed by an offset time, and a rest event is characterized by an offset time followed by a corresponding onset time.

[0035]

[0040] The embodiment transmits the generated HP event data 117 to the HP event store 153. For example, the haptic event processor 115 may include a local buffer for recording detected haptic events, and the haptic event processor 115 outputs the HP event data 117 as one or more tracks of a sequence of HP events and / or a sequence of HP onset time and HP offset time. In some implementations, the HP event store 153 is implemented in the device I / O subsystem 105, such as being integrated with the haptic event processor 115. In other implementations, the HP event store 153 is implemented in the rhythm evaluator subsystem 150, as shown in the figure. In other implementations, the HP event store 153 is implemented as part of a storage subsystem (not shown) that includes the HP event store 153 and an MS data store 140. The HP event store 153 can be implemented using any suitable data storage, such as solid-state storage, hard disk storage, removable media storage, or network storage (e.g., local network, cloud-based, etc.). Furthermore, the HP event data 117 can be stored by the HP event store 153 in any suitable manner, such as a flat file data structure or a hierarchical data structure.

[0036]

[0041] Embodiments of the rhythm evaluator subsystem 150 evaluate the user's performance (i.e., HP event data 117 acquired and stored in the HP event store 153) based on how accurately the rhythm performed by the user corresponds to the rhythm represented by the sequence of SNR events indicated in the musical score. Embodiments of the rhythm evaluator subsystem 150 include an evaluation engine 155 and a feedback engine 157. The evaluation engine 155 generates evaluation data based on comparing performance data from the HP event store 153 with SNR data represented by MS logical data 143. The evaluation data is used by the feedback engine 157 to generate feedback data 159. The feedback data 159 may include micro-level feedback and macro-level feedback. The feedback data 159 may be used by the display processor 125 to generate graphic performance feedback output to the user via the display interface 120.

[0037]

[0042] In some embodiments, the evaluation engine 155 performs its evaluation by iterating through each of the sequences of SNR events. For each SNR event, the evaluation engine 155 finds a matching HP event (i.e., an HP event representing a user's attempt to perform the corresponding SNR event) and attempts to determine how accurately the matching HP event matches the rhythm of the corresponding SNR event (e.g., its SNR onset time and / or SNR offset time). The timing of the HP events is recorded with reference to a performance time criterion. For example, each HP onset time and HP offset time is represented by a timestamp that is a certain time distance (e.g., in milliseconds) from the start of the performance. The performance time criterion may be the real-time reference criterion described above, or any other suitable time criterion. As described above, some embodiments include score-referenced MS logical data 143. Such embodiments can convert the score-referenced MS logical data 143 into a performance time criterion. For example, score-referenced MS logical data 143 is converted to real-time referenced MS logical data 143, and if necessary, the real-time referenced MS logical data 143 is further temporally aligned with respect to a performance time criterion (or vice versa).

[0038]

[0043] The evaluation engine 155 can obtain a set of evaluation criteria for each SNR event. The evaluation criteria can be formulated as processor-readable evaluation functions such as if-then statements and matching functions. Some or all of the evaluation criteria can be stored as part of the MS logical data 143 of the SNR event, and / or some or all of the evaluation criteria can be derived based on predetermined thresholds and / or rules. One type of evaluation criterion is a "valid match window". A valid match window is a time window that references the SNR onset time of each SNR event. In some implementations, the valid match window is symmetric with respect to the SNR onset time, such as a 100-millisecond window extending 50 milliseconds on both sides of the SNR onset time. In other implementations, the valid match window is asymmetric, for example, a 100-millisecond window that starts 25 milliseconds before the SNR onset time and ends 75 milliseconds after the SNR onset time. In some implementations, the valid match window is referenced in real time with a predetermined fixed size. For example, the valid match window is always 100 milliseconds and is always symmetric with respect to the SNR onset time of all SNR events. In some implementations, the valid match window is score-referenced and rule-based. In one such implementation, the valid match window is defined as a function of the beat (e.g., a ratio or percentage, or some formulaic relationship). For example, the valid match window shrinks at faster tempos. In another such implementation, the valid match window is defined as a function of the user's skill level and / or the difficulty of the music score. For example, the valid match window is more lenient towards less advanced players and / or simpler scores.

[0039]

[0044] As described above, the evaluation engine 155 can find a matching HP event (representing the user's performance of the SNR event) for each SNR event. Embodiments of the evaluation engine 155 can search for HP events in the HP event store 153 to determine if any of the retrieved HP events are a “matching event,” meaning that the HP event has an HP onset time that falls within a valid match window for the SNR event. In some implementations, if multiple HP events are found to have HP onset times within a valid match window, the evaluation engine 155 selects the HP event with the HP onset time closest to the SNR onset time as the matching event. Some embodiments iterate through all SNR events in the MS data store 140 to find a matching HP event. Other embodiments may ignore some of the SNR events in the MS data store 140. For example, if a musical score includes certain types of musical groupings (e.g., quintuplets) and / or certain expressive style specifications (e.g., tempo rubato passages), SNR events related to these parts of the musical score may be ignored or treated differently.

[0040]

[0045] For each evaluated SNR event, the evaluation engine 155 either finds a matching HP event or does not. If no matching HP event is identified (i.e., no HP event with an HP onset time exists within the valid match window of the SNR event), the SNR event can be tagged as a “skipped event”. If a matching HP event is identified (i.e., an HP event with an HP onset time exists within the valid match window of the SNR event), the matching HP event can be associated with the SNR event. Such tagging can be performed in any suitable way, such as by setting a flag associated with the SNR event or by storing a label associated with the SNR event. In some embodiments, the rhythm evaluator subsystem 150 includes an evaluation data store 156 for storing evaluation results. For example, additional storage can be used to store an instance of each evaluated SNR event along with its generated evaluation data (e.g., tags, labels, etc.), and / or the evaluation data along with a pointer or other identifier linking the evaluation data to the corresponding SNR event stored in the MS data store 140.

[0041]

[0046] If a matching HP event is identified (i.e., an HP event with an HP onset time exists within the valid match window of the SNR event), the matching HP event can be associated with the SNR event. Each matching event (or matching HP event) is one of the HP events that has been determined to be temporally corresponding to one of the SNR events. Such association can be performed by storing a pointer between the matching HP event and the evaluated SNR event, or by any other appropriate method. Each matching event is one of the HP events that has been determined to be temporally corresponding to one of the SNR events.

[0042]

[0047] For each matching HP event, the evaluation engine 155 can then evaluate the rhythmic fit between the matching HP event and its temporally corresponding SNR event. The rhythmic fit may be based on one or more of at least three rhythmic properties of the SNR event: onset fit, such as between HP onset time and SNR onset time; offset fit, such as between HP offset time and SNR offset time; and / or duration fit, such as between the duration of the HP event (i.e., the temporal distance between HP onset time and HP offset time) and the duration of the SNR event (i.e., the temporal distance between SNR onset time and SNR offset time). In some embodiments, the rhythmic fit between a matching HP event and its temporally corresponding SNR event is expressed as a score functionally related to the temporal distance between the properties. For example, a high onset fit score indicates a short temporal distance between HP onset time and SNR onset time. The scoring may differ for each different property. In other embodiments, the rhythmic fit between a matching HP event and its temporally corresponding SNR event is expressed categorically based on one or more predetermined thresholds. For example, onset suitability is determined as "good" or "poor" (or "high," "medium," and "low," etc.) by determining whether the temporal distance between the HP onset time and the SNR onset time is greater than or less than one or more predetermined thresholds. One or more predetermined thresholds may differ for different characteristics.

[0043]

[0048] As described above, the evaluation engine 155 can obtain a set of evaluation criteria for each SNR event. The evaluation criteria may include setting an effective window, an evaluation threshold, scoring, and / or other parameters for evaluating rhythmic fit. In some embodiments, the evaluation criteria are generated based on predetermined fixed settings. For example, as described above, the effective match window may be defined as a fixed time window relative to the SNR onset time. In other embodiments, some or all of the evaluation criteria are rule-based. Using such rule-based evaluation criteria, the evaluation of rhythmic fit can be adjusted based on tempo, the user's skill level, the difficulty of the musical score or passage, the SNR event type, and / or other additional information. For example, the effective match window may be adjusted so that matching HP events are identified more or less strictly according to tempo, note type, etc.

[0044]

[0049] As another example, the evaluation criteria could be set so that the rhythmic fit of note SNR events is evaluated based on stricter onset fit and stricter offset fit, and the rhythmic fit of rest SNR events is evaluated based on stricter onset fit and stricter offset fit. As yet another example, the evaluation criteria can be used to set an effective offset window. An effective offset window can define a window for evaluating offset fit with respect to the HP offset time of a matching HP event, and / or for evaluating duration fit with respect to the duration of the HP event. For example, an effective offset window could determine an appropriate HP offset time and / or HP event duration as a temporal distance from the HP onset time, based on whether the corresponding SNR event is staccato, legato, a rest, etc.

[0045]

[0050] The evaluation criteria may also be formulated and applied by the rhythm evaluator subsystem 150 to handle special cases. One category of special cases involves musically relevant grouping of SNR events. An example of musically relevant grouping is a chord. In some cases, the rhythm evaluator subsystem 150 evaluates a chord as a single SNR event. For example, a chord can be stored in a single set of MS logical data 143, or treated as a single event having a single SNR onset time, a single SNR offset time, a single duration, etc. In other cases, such as obtaining multiple simultaneous tracks of HP events representing multiple fingers, etc., using the multi-touch haptic interface 110, each note of a chord can be treated as a separate SNR event. For example, in some musical pieces, one or more notes of a chord are held and other notes are released (e.g., represented by using one stem direction for one or more held notes and another stem direction for one or more moving notes).

[0046]

[0051] Another example of musically relevant groupings is a "quintuplet" (a group of five sixteenth notes to be performed within the span of a single quarter note), or any other grouping of several notes to be played on a defined number of beats. One implementation may treat each note in the group as any other note, each having its own SNR onset time and SNR offset (and / or SNR event duration) so as to be evenly spaced and to precisely consume the fifth note of each quarter note duration. In such an implementation, each note may be evaluated for rhythmic relevance by an evaluation engine 155, for example, to find a matching HP event. Other implementations may treat the entire group as an SNR event (e.g., instead of, or in addition to, storing each component of the group as an SNR event). In one such implementation, the evaluation engine 155 can find a suitable set of constituent HP events (e.g., HP events for five notes) as a matching event by determining whether the set of constituent HP events starts within a valid match window of the grouped SNR event and whether all of them fall within a valid offset window of the grouped SNR event (e.g., corresponding to the total duration of the quintet). A similar approach can be applied to other types of musically relevant groupings, such as the handling of ornaments, trills, mordents, glissandos, etc.

[0047]

[0052] Another category for special cases includes expressive notation. One example is a passage marked "tempo rubato" in the score, indicating that it should be played freely. In some cases (for example, for certain genres, composers, etc.), such a tempo rubato passage may indicate that the performer is completely free from rhythmic constraints. In some such cases, embodiments may set evaluation criteria to ignore SNR events that come into play within the tempo rubato section when performing rhythmic evaluation. For example, these SNR events may be specially tagged or labeled. In other such cases, the evaluation engine 155 uses pattern matching to determine which set of HP events represents the user's performance in the tempo rubato passage in order to correct subsequent timing. Such pattern matching may include counting the number of HP events corresponding to the number of SNR events in the passage, or attempting to match the relative duration of the HP events to the different durations of the SNR events in the passage. By using such a pattern matching approach, the evaluation engine 155 can shift the post-passage performance time criterion (i.e., shift the timing of the post-tempo rubato passage) so that HP events can properly match the temporally corresponding SNR events after the tempo rubato passage, regardless of how much time the performer spent playing the passage.

[0048]

[0053] In other cases, even if a passage is marked as rubato, certain rhythmic constraints are expected. For example, performing a tempo rubato section of a particular piece may involve strictly adhering to the rhythm with the left hand while playing more rhythmically and freely with the right hand. As another example, some composers or genres intend for a tempo rubato passage to be performed over a fixed overall duration (e.g., consuming a total number of seconds, beats, or measures, etc.), even if individual notes within the passage may be accelerated or decelerated in a personal and expressive way. In such cases, the embodiment may set stricter evaluation criteria for rhythmic evaluation at the passage level, while setting looser evaluation criteria or no evaluation criteria at the note-level or rest-level. In one such case, the evaluation criteria are defined so that the evaluation engine 155 can search for a set of matching HP events, where the first of the matching HP events has good onset fit with the SNR onset time of the first SNR event in the passage, the last of the matching HP events has good offset fit with the SNR offset time of the last SNR event in the passage, and the number of HP events in the set of HP events is within a predetermined threshold amount (e.g., number, percentage, etc.) of the total number of SNR events in the passage. In another such case, the evaluation criteria include further definitions of note-level or rest-level. For example, the evaluation criteria look for onset fit for each SNR event in the passage to determine whether any of the matching HP events were executed outside a predetermined tolerance range (e.g., the HP onset time is too long before or after the corresponding SNR onset time). A similar approach can be applied to other types of expressive notation, such as the handling of fermatas.

[0049]

[0054] Based on the above, the evaluation engine 155 uses evaluation criteria to generate evaluation data that shows the rhythmic fit between at least the user's performance and a sequence of SNR events (e.g., all SNR events in a musical score, or an evaluated portion of an SNR event). In some embodiments, the evaluation data is stored in the evaluation data store 156. Embodiments of the feedback engine 157 can then use the evaluation data (e.g., from the evaluation data store 156) to generate and output feedback data 159. In some embodiments, the feedback data 159 is generated so that it can be used by the display processor 125 to generate graphic performance feedback that is output to the user via the display interface 120. In some embodiments, the feedback engine 157 generates further feedback data 159 that can be used by the audio processor 135 to generate audible performance feedback that is output to the user via the audio interface 130.

[0050]

[0055] In some embodiments, the feedback data 159 is micro-level feedback indicating event-by-event rhythmic fit as a graphic overlay on the musical score. The term “graphic overlay” is used herein to generally include any type of simultaneous graphic display that provides a visual juxtaposition between the graphic feedback element and the graphic score element to which the feedback is applied. For example, such a graphic overlay could include displaying the graphic feedback element semi-transparently on top of the graphic score element, recoloring the graphic score element to imply the graphic feedback element (e.g., changing the line thickness, etc.), or adding text or images representing the graphic feedback element together with the graphic score element. In one implementation, SNR events are displayed on the musical score in a first color to indicate skipped events (i.e., no matching HP event was found), a second color to indicate successful performances (e.g., a matching HP event was found and the matching HP event has good onset, offset, and / or duration fit), and a third color to indicate unsuccessful performances (e.g., a matching HP event was found, but the matching HP event has poor onset, offset, and / or duration fit). In another implementation, the first color is used to highlight the portion of the musical score where an SNR event should be played (e.g., from the SNR onset time to the SNR offset time of a particular note's SNR event), the second color is used to highlight the portion of the musical score where a matching HP event was played well, and the third color is used to highlight the portion of the musical score where a matching HP event was played poorly. In yet another implementation, each SNR event is labeled with a score indicating the rhythmic fit evaluated for that SNR event. In yet another implementation, colors or other graphic elements are further used to overlay historical performance data from the user or other performers.

[0051]

[0056] In other embodiments, the feedback data 159 is macro-level feedback indicating rhythmic conformance for each performance. The macro-level feedback may be presented as a graphic overlay on the musical score in a separate part of the display or in any suitable way. In some such embodiments, the macro-level feedback shows an overall performance score (e.g., in text format and / or any other suitable format). For example, the feedback data 159 shows that the performance had an overall rhythmic conformance of 82%. In other such embodiments, the feedback engine 157 performs one or more statistical analyses on the evaluation data to look for patterns or trends, and the feedback data 159 shows the results of those analyses. One such analysis shows a pattern of performance across passages and / or sections of the performance. For example, the feedback data 159 shows that the user's performance was good (i.e., had high rhythmic conformance) in measures 1-10, very bad in measures 11-13, and moderately good for the rest of the performance. Another such analysis matches the performance pattern to a predetermined performance category. For example, feedback data 159 indicates that performance was inadequate with rests rather than notes (e.g., this analysis compares the rhythmic correspondence of rest SNR events with that of non-SNR events). Another example is that feedback data 159 indicates an overall tendency to play before or after the beat (e.g., the analysis revealed a statistical tendency for HP onset times to be earlier than SNR onset times or delayed relative to SNR onset times). Yet another example is that feedback data 159 indicates a misunderstanding of certain musical notation (e.g., the analysis revealed that two HP events existed even when two notes were linked together, indicating a misunderstanding of cohesion; and all HP events corresponding to triplets appeared to have the duration and interval of eighth notes, indicating a misunderstanding of triplets).As another example, feedback data 159 indicates that the performer appears to be missing parts of the musical score (for example, analysis reveals that HP events are missing for an entire hand in a two-handed piece, or for an entire section of a piece). As yet another example, feedback data 159 indicates that the performer appears to be struggling with certain types of passages (for example, analysis reveals that rhythmic correspondence tends to be higher between fast or slow sections of a piece).

[0052]

[0057] The above assumes that rhythm evaluation is performed in the direction of SNR events, such as by attempting to match user-performed HP events to SNR events in the score. For example, each SNR is evaluated to find a matching HP event, or the event is considered skipped. Some embodiments of the rhythm evaluator subsystem 150 perform further and / or alternative evaluations in the direction of HP events. In some such embodiments, HP events are analyzed to look for potential interface errors. For example, a glitch in the haptic interface 110, or a slight movement of the user's hand or finger on the haptic interface 110, may cause a certain type of haptic event processor 115 to record multiple HP events, even if only a single HP event was intended by the performer. Some implementations can address such cases in preprocessing. The haptic event processor 115 can detect when the HP offset time of one HP event and the HP onset time of the next HP event are close to a predetermined threshold, indicating that the user is unlikely to have intended separate events. The haptic event processor 115 can then treat these events as a single HP event (e.g., by generating a single event in the HP event data 117 and removing intermediate offset and onset time data). Other implementations can handle such cases in post-processing. For example, the evaluation engine 155 can detect groups of consecutive HP events (e.g., 2, 3, etc.) where the first HP event has high onset compatibility with the SNR onset time of the SNR event, the last HP event has high offset compatibility with the SNR offset time of the same SNR event, and the group of consecutive HP events has the smallest interval between them. The removal of unintended HP event data in these and other cases can be called "deduplicating".

[0053]

[0058] In other such embodiments, HP events are analyzed (e.g., by evaluation engine 155) to evaluate non-matching (non-deduplication) HP events. For example, there may be more HP events than SNR events, but some or all of the irrelevant HP events appear to have been intentionally performed by the user. Some embodiments of evaluation engine 155 analyze these irrelevant HP events to determine whether they represent a misunderstanding of a particular notation rule (e.g., note bundling), whether they suggest an error in processing haptic information, or whether the user appears to have been improvising or riffing for a period of time. In another example, evaluation engine 155 may determine that the number of SNR events identified as skipped events is similar to the number of HP events not identified as matching events (e.g., an entire section of an HP event may have been performed correctly in itself but somehow time-shifted relative to an SNR event). Embodiments of evaluation engine 155 can identify such cases and / or attempt to improve such cases (e.g., by attempting to map non-matching HP events to missed SNR events using a moving window or other pattern matching technique).

[0054]

[0059] Figure 2 shows another exemplary haptic rhythm training system 200 implemented in a cloud-based environment according to various embodiments described herein. The system 200 includes one or more user devices 210 that communicate with one or more remote servers 230 via one or more communication networks 240 (only a single instance of each is shown to avoid overcomplicating the figure). Each user device 210 may implement an instance of the device I / O subsystem 105, a thin client 215, and a network interface 220, respectively. One or more remote servers 230 implement a rhythm evaluator subsystem 150 and a remote storage subsystem 235. The remote storage subsystem 235 may include some or all of the MS datastore 140, the HP event store 153, and the evaluation datastore 156. For example, the remote storage subsystem 235 can be used to maintain a record of historical performance data.

[0055]

[0060] For example, during operation, the user accesses an application that provides functionality for the rhythm evaluator subsystem 150 by communicating over one or more networks 240 using a thin client 215 of the user device 210. The operation of the thin client 215 can include the communication of various types of data. For example, MS visual data 145 and MS logical data 143 are received by the user device 210 from the remote storage subsystem 235 of one or more remote servers 230, HP event data 117 are sent back from the user device 210 to the rhythm evaluator subsystem 150 of one or more remote servers 230, and feedback data 159 are received by the user device 210 from the rhythm evaluator subsystem 150 of one or more remote servers 230. Communication between the user device 210 and one or more remote servers 230 over one or more networks 240 is facilitated by a network interface 220. For example, one or more networks 240 may include any suitable wired or wireless link to any one or more public and / or private networks, local and / or remote networks, etc., and the network interface 220 may implement (or facilitate the use of) any relevant ports, protocols, etc.

[0056]

[0061] Figures 3A to 3C show exemplary screenshots 300 from exemplary applications for implementing features of a haptic rhythm training system according to various embodiments described herein. Referring to Figure 3A, the first screenshot 300a shows a graphics performance interface having several areas or sections. Figures 3A to 3C assume a computing environment in which a touchscreen display implements a display interface 120 for graphics output and a haptic interface 110 for haptic input. Static or dynamic graphics output to the display interface 120 is generated by the display processor 125, and any haptic interaction with the haptic interface 110 (e.g., the user tapping or holding a finger or hand on the touchscreen) is detected by the haptic event processor 115 and converted into HP event data 117.

[0057]

[0062] As illustrated, the first part of the graphics performance interface (first interface area 310) displays a musical score with a musical notation representation of SNR events. In the illustrated example, the first interface area 310 displays a simplified piano arrangement of a portion of the "Ode to Joy" from Beethoven's Symphony No. 9. The displayed musical score includes a grand staff with treble and bass clefs, notation of notes and rests (corresponding to the sequence of SNR events), notation of time signatures, notation of bar boundaries, notation of fingerings for the right and left hands, and so on.

[0058]

[0063] A second portion of the graphics performance interface (second interface area 320) is for haptic input. As described herein, embodiments can implement the second interface area 320 in different ways. In some implementations, only a single track of HP events is received, so that only a single haptic interface area is required. Some such implementations may include a single designated area, such as a single area with graphically defined boundaries for accessing haptic interactions that are detected as HP events. For example, the haptic event processor 115 is configured so that only user interactions within a designated area of ​​the haptic interface 110 are used to generate HP event data 117. Other such implementations do not graphically designate any particular area, and any haptic interaction with any part of the haptic interface 110 is detected as an HP event by the haptic event processor 115.

[0059]

[0064] In other implementations, multiple tracks of HP events are received simultaneously, requiring multiple haptic interface sub-regions. Some such implementations include explicit graphical definitions of multiple sub-regions, such as by displaying graphically defined boundaries. In the illustrated example, the second interface region 320 includes two explicitly designated sub-regions, one labeled "L" for left-hand haptic input and the other labeled "R" for right-hand haptic input. Any haptic interaction within the first sub-region of the haptic interface 110 is detected by the haptic event processor 115 as a left-hand HP event, and any haptic interaction within the second sub-region of the haptic interface 110 is detected by the haptic event processor 115 as a right-hand HP event. Other implementations can specify any appropriate number of sub-regions. For example, four sub-regions can be specified to practice the rhythm of a four-part moving harmony, or ten sub-regions can be specified to practice using all ten fingers. Implementations can specify sub-regions in any appropriate way. For example, one of two sub-regions can be graphically designated as the left third toe of the touchscreen, the other as the right third toe of the touchscreen, or ten sub-regions can be graphically designated to resemble two sets of five fingers or two sets of five circles. Other such implementations do not involve explicit graphical designation of sub-regions. Rather, the haptic event processor 115 can group haptic inputs by position such that detected interactions with the haptic interface 110 occurring within a threshold distance of each other are grouped into a single track of HP events. For example, during a particular practice, the user's right thumb consistently touches the touchscreen in a 9-square-inch area near the lower right region of the display, and the user's left thumb consistently touches the touchscreen in a 6-square-inch area near the center-left region of the display, so the haptic event processor 115 can identify the interactions between the right and left thumbs as separate, simultaneous tracks of HP events.

[0060]

[0065] In some embodiments, the first interface area 310 and the second interface area 320 do not completely overlap, and no part of the second interface area 320 overlaps with any part of the first interface area 310. In other embodiments, the first interface area 310 and the second interface area 320 partially overlap. In other embodiments, the first interface area 310 and the second interface area 320 completely overlap. For example, the first interface area 310 and the second interface area 320 consume the same area, and the second interface area 320 is entirely within the first interface area 310, or the first interface area 310 is entirely within the second interface area 320.

[0061]

[0066] In some embodiments, the first interface area 310 is an output-only area, and haptic interactions with the first interface area 310 are ignored by the haptic event processor 115. In one such embodiment, only haptic interactions within the second interface area 320 (separate from the first interface area 310) are detected by the haptic event processor 115. In another such embodiment, the graphics performance interface includes one or more additional areas other than the first interface area 310 and the second interface area 320 where haptic interactions are detected and processed, but not for generating HP event data 117. For example, other haptic interactions in other areas of the graphics performance interface may be detected for interface navigation or control, such as pausing or resuming playback, ending practice, accessing one or more menus, or switching applications. In other embodiments, haptic interactions with the first interface area 310 are not used for HP event generation but are still detected by the haptic event processor 115 to support other functions. For example, tapping the first interface area 310 may pause or resume playback, and double-tapping the first interface area 310 may open a menu.

[0062]

[0067] During operation, as described herein, the output in the first interface area 310 is used to provide at least a graphical representation of real-time progress through the musical score at tempo. Real-time progress can be indicated by a cursor or other graphical element 315 indicating the current playback position 315 on the score mapped to the current playback time at tempo. As illustrated, the graphical element may include a box that highlights one of the currently playing SNR events so that it is graphically displayed on the musical score. Alternatively, the same type of highlighting box may indicate the current beat of playback (for example, all SNR events that fall within that beat are within the highlighted box). Additionally or alternatively, the graphical element may include a vertical line that moves horizontally across the measures of the score at a speed corresponding to the playback tempo. While the graphical representation is output to the first interface area 310, HP events are captured in the second interface area 320. HP events represent the user's rhythmic performance of the musical score at tempo as it is being played at tempo.

[0063]

[0068] Referring to Figure 3B, the second screenshot 300b shows the graphic performance after several user haptic interactions have been captured (e.g., during a practice performance). For example, the current playback position 315 is shown to be on the first beat of the last measure of the score. During the user's performance, real-time graphic feedback can be provided to show the HP event data 117 being generated by the haptic interactions performed by the user. In the illustrated example, highlighting (e.g., a different color from the color indicating the current playback position 315) shows the captured HP events overlaid on the graphic representation of the SNR events. Each highlighting may show a graphic representation of the respective HP onset time 330 (i.e., corresponding to the time when the user began interacting with the haptic interface 110), a graphic representation of the respective HP offset time 331 (i.e., corresponding to the time when the user stopped interacting with the haptic interface 110), and / or a graphic representation of the respective HP event duration 332 (i.e., corresponding to the amount of time the user maintained interaction with the haptic interface 110). The performer can understand, for example, that not all quarter notes are held for the same amount of time. This diagram also shows that some SNR events are not overlaid on HP event data 117, and these SNR events appear not to have been performed by the user.

[0064]

[0069] Referring to Figure 3C, the third screenshot 300c shows the graphic performance after the performance has been completed and evaluated by the rhythm evaluator subsystem 150. The current playback position 315 is not shown because the music score is no longer being played. Visual feedback 340 is provided to graphically map the evaluation data (generated by the rhythm evaluator subsystem 150) to the musical notation representation of SNR events on the music score. In the illustrated example, the first highlighting color is used to indicate events that were skipped as SNR events that were detected as not having temporally corresponding HP event data, and at least the second highlighting color is used to indicate events that were performed as SNR events that were detected as having temporally corresponding HP event data (for example, the highlighting can further distinguish between successful and unsuccessful events).

[0065]

[0070] As described herein, several embodiments can provide both micro-level and macro-level feedback. In the illustrated example, visual feedback 340 is used to provide micro-level feedback. For example, visual feedback 340 indicates a rhythm performance evaluation for individual SNR elements (e.g., and / or groups of SNR elements). In some embodiments, a feedback area 350 is provided to display macro-level feedback. In the illustrated example, the feedback area 350 is replaced by a second interface area 320. For example, the rhythm evaluator subsystem 150 determines that the user did not perform a section of the score (e.g., there is a section with multiple consecutive missed events) and generates the corresponding macro-level feedback "It seems you stopped playing in the middle of the song. Try again and give it your all!". Similarly, the rhythm evaluator subsystem 150 may determine that the user did not perform any of the score (e.g., no HP events are detected throughout the entire performance) and generate the corresponding macro-level feedback "It seems you decided not to perform!".

[0066]

[0071] Some embodiments include additional types of interfaces for providing the user with past feedback and / or other information. For example, Figures 6A and 6B show exemplary screenshots 600 from exemplary applications for implementing a history user feedback function of a haptic rhythm training system according to various embodiments described herein. In the illustrated examples, the display interface includes a history feedback control area 610, which includes a slider bar. The slider bar shows past performance times. By sliding to different positions on the slider bar, the user can access feedback from each of those performance times. For example, screenshot 600a of Figure 6A shows a performance at 09:54 on August 12, and screenshot 600b of Figure 6B shows a performance four minutes later. Each performance yielded different feedback. Such past feedback data can be stored in local storage (e.g., the system in Figure 1), remote storage (e.g., the system in Figure 2), or any suitable method. Such past performance data can be displayed to the user in an alternative way. Some implementations overlay history data from multiple performances. Some implementations generate trend data (e.g., statistics and / or other information) based on past performances. Some such implementations generate and provide macro-level feedback based on trend data across historical performance. Some such implementations suggest additional practices based on such trend data.

[0067]

[0072] Embodiments of the haptic rhythm training system or its components can be implemented and / or incorporated on one or more computer systems, as shown in Figure 4. Figure 4 provides a schematic diagram of one embodiment of a computer system 400 capable of implementing various system components and / or performing various steps of the methods provided by various embodiments. It should be noted that Figure 4 is intended only to provide a generalized description of the various components, any or all of which may be appropriately utilized. Thus, Figure 4 broadly illustrates how individual system elements may be implemented in a relatively isolated or relatively more integrated manner.

[0068]

[0073] The computer system 400 is shown to include hardware elements that can be electrically coupled (or may communicate as needed) via a bus 405. The hardware elements may include one or more processors 410, including, but not limited to, one or more general-purpose processors and / or one or more dedicated processors (such as a digital signal processing chip, a graphics accelerator, or a video decoder). As illustrated, some embodiments include a device I / O subsystem 105, which may include one or more I / O devices 415 and one or more I / O processors 417. As described herein, the I / O device 415 may include a haptic interface 110, a display interface 120, and an audio interface 130. The I / O processor 417 may include a haptic event processor 115, a display processor 125, and an audio processor 135. Furthermore, input devices may include, but are not limited to, buttons, knobs, switches, keypads, touchscreens, remote controls, etc., and output devices may include, but are not limited to, displays, indicators, gauges, etc. Some embodiments of the computer system 400 interface with additional computers, peripheral devices, etc., so that the device I / O subsystem 105 can include various physical and / or logical interfaces (e.g., ports) to facilitate interaction and control between components.

[0069]

[0074] The computer system 400 may further include (and / or communicate with) one or more non-temporary storage devices 425, which may include, but are not limited to, local and / or network-accessible storage, and / or may include, but are not limited to, solid-state storage devices such as disk drives, drive arrays, optical storage devices, random access memory ("RAM"), and / or read-only memory ("ROM"), which may be programmable, flash-updatable, etc. Such storage devices may be configured to implement appropriate data stores, which include, but are not limited to, various file systems, database structures, etc. In some embodiments, the storage device 425 includes non-temporary memory 240. In some embodiments, the storage device 425 may include an MS data store 140, an HP event store 153, and / or an evaluation data store 156.

[0070]

[0075] The computer system 400 may also include a communications subsystem 430 which may include, but is not limited to, any suitable antenna, transceiver, modem, network card (wireless or wired), infrared communication device, wireless communication device, chipset (e.g., Bluetooth® device, 802.11 device, WiFi device, WiMAX device, cellular communication device, etc.), and / or other communications components. As shown in the figure, the communications subsystem 430 may also include a network interface 220 to facilitate communication between a user device and a remote server over a communications network. The communications subsystem 430 can further facilitate communication with other computing systems.

[0071]

[0076] In many embodiments, the computer system 400 further includes working memory 435 which may include RAM or ROM devices as described herein. The computer system 400 may also include software elements indicated as currently located in working memory 435, including an operating system 440, device drivers, executable libraries, and / or computer programs provided by various embodiments, and / or other code such as one or more application programs 445 which may be designed to implement methods provided by other embodiments and / or configure the system as described herein. As mere examples, one or more procedures described with respect to the methods described herein may be implemented as code and / or instructions executable by a computer (and / or a processor in the computer), and in some embodiments such code and / or instructions may be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations in accordance with the methods described herein. In some embodiments, the operating system 440 and working memory 435 are used together with one or more processors 410 to implement the functions of the rhythm evaluator subsystem 150.

[0072]

[0077] These instructions and / or sets of code may be stored in a non-temporary computer-readable storage medium, such as the non-temporary storage device 425 described above. In some cases, the storage medium may be incorporated into a computer system, such as computer system 400. In other embodiments, the storage medium may be separate from the computer system (e.g., a removable medium such as a compact disk) and / or provided in an installation package, and the storage medium may be used to program, configure, and / or adapt a general-purpose computer on which the instructions / code are stored. These instructions may take the form of executable code that can be executed by computer system 400, and / or take the form of source and / or installable code when compiled and / or installed on computer system 400 (e.g., using various commonly available compilers, installation programs, compression / decompression utilities, etc.), and then take the form of executable code.

[0073]

[0078] It will be apparent to those skilled in the art that substantial modifications can be made to suit specific requirements. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0074]

[0079] As described above, in one aspect, several embodiments may employ a computer system (such as computer system 400) to carry out the methods according to various embodiments of the present invention. According to a series of embodiments, some or all of the steps of such a method are executed by computer system 400 in response to processor 410 executing one or more sequences of one or more instructions contained in working memory 435 (which may be incorporated into other code such as the operating system 440 and / or application program 445). Such instructions may be read into working memory 435 from another computer-readable medium such as one or more non-temporary storage devices 425. As just one example, the execution of a sequence of instructions contained in working memory 435 can cause processor 410 to carry out one or more steps of the method described herein.

[0075]

[0080] As used herein, the terms “machine-readable medium,” “computer-readable storage medium,” and “computer-readable medium” refer to any medium involved in providing data that enables a machine to operate in a particular manner. These mediums may not be transient. In embodiments implemented using the computer system 400, various computer-readable mediums may be involved in providing instructions / code to the processor 410 for execution and / or may be used to store and / or carry such instructions / code. In many implementations, the computer-readable medium is a physical and / or tangible storage medium. Such a medium may take the form of a non-volatile medium or a volatile medium. Non-volatile mediums include, for example, optical and / or magnetic disks such as the non-temporary storage device 425. Volatile mediums include, but are not limited to, dynamic memory such as the working memory 435. Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or other magnetic media, CD-ROMs, other optical media, other physical media with mark patterns, RAM, PROMs, EPROMs, FLASH-EPROMs, other memory chips or cartridges, or other media from which a computer can read instructions and / or code. Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 410 for execution. Simply as an example, the instructions may first be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions as signals over a transmission medium that is received and / or executed by the computer system 400. The communication subsystem 430 (and / or its components) generally receives the signals, and the bus 405 may then carry the signals (and / or the data, instructions, etc., carried by the signals) to the working memory 435, from which the processor 410 retrieves and executes the instructions.Instructions received by the working memory 435 may optionally be stored in the non-temporary storage device 425 either before or after execution by the processor 410.

[0076]

[0081] Furthermore, it should be understood that the components of computer system 400 can be distributed across the entire network. For example, some processes may be executed in one location using a first processor, while other processes may be executed by another processor located away from the first processor. Other components of computer system 400 can be distributed in a similar manner. Therefore, computer system 400 can be interpreted as a distributed computing system that performs processing in multiple locations. In some cases, computer system 400 may be interpreted as a single computing device, such as a separate laptop or desktop computer, depending on the context.

[0077]

[0082] Figure 5 shows a flowchart of exemplary Method 500 for haptic-based evaluation of rhythmic performance referencing a musical score, in various embodiments. Method 500 can be carried out using any suitable system, including those described above in Figures 1 to 4. Embodiments of Method 500 begin in Stage 504 by outputting an audiovisual representation of real-time progress through a musical score at a given tempo via the output device interface of a computing device. The musical score includes a musical notation representation of each of a sequence of SNR events. Each SNR event has a stored logical definition including an associated SNR onset time and an associated SNR offset time. In some embodiments, some of the SNR events are note events and others are rest events. The musical notation representation of each note event indicates one or more notes of one or more pitches to be held over a notated duration, beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event. The musical notation for each rest event indicates silence that should be maintained over a notated duration, beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event.

[0078]

[0083] In some embodiments, the output step in stage 504 includes the steps of: displaying a musical score via an output device interface; playing audio corresponding to at least each of the sequences of SNR events to audibly represent the real-time progress of the musical score at a certain tempo; and, during playback, displaying a graphic position indicator (e.g., a cursor, a box around the currently playing SNR event) overlaid on the musical score, which dynamically updates to indicate which of the sequences of SNR events corresponds to the audio currently being played, so that the graphic position indicator visually represents the real-time progress of the musical score at tempo. In some such embodiments, the output step in stage 504 further includes, during playback, displaying a graphic representation of the associated HP onset time and associated HP offset time of each of a plurality of HP events as each of the HP events is acquired, overlaid on the musical score. For example, such an overlay provides the user with real-time feedback indicating how the haptic interaction is being received (e.g., when the HP onset time and HP offset time are recorded for score playback). In some such embodiments, the output step in stage 504 further includes a step of playing a rhythm track (e.g., a metronome, backing track, etc.) in sync with the playback of the audio at tempo.

[0079]

[0084] In stage 508, the embodiment can acquire HP events corresponding to user interaction with the computing device's haptic device interface during the output step. The HP events represent the user's rhythmic performance of the musical score at tempo. Each HP event has an associated HP onset time and an associated HP offset time.

[0080]

[0085] In stage 512, the embodiment can generate evaluation data for SNR events by a rhythm evaluator subsystem that communicates with an output device interface and a haptic device interface. The generation step in stage 512 may include steps to perform stages 516 and 520. In stage 516, the embodiment can obtain relevant evaluation criteria for each SNR event based on the relevant SNR onset time and the relevant SNR offset time. In stage 520, the embodiment can compare the relevant evaluation criteria for each SNR event with the relevant HP onset time and the relevant HP offset time for at least several HP events. The comparison step in stage 520 may include identifying matching events for HP events that are determined to be temporally corresponding to one of the SNR events, and evaluating the rhythmic compatibility between each of the multiple matching events and the temporally corresponding SNR events.

[0081]

[0086] In some embodiments, the output step in stage 508 establishes a performance time criterion such that each HP onset time and each HP offset time corresponds to the respective time values ​​of the performance time criterion. In such embodiments, the step of generating evaluation data in stage 512 may include mapping the associated SNR onset time and associated SNR offset time to the performance time criterion for each SNR event. The evaluation criterion in stage 516 can be defined against the performance time criterion, and the comparison step in stage 520 can be performed against the performance time criterion.

[0082]

[0087] In some embodiments, the step of obtaining relevant evaluation criteria in stage 516 includes the step of calculating an effective match window for each SNR onset time. In some such embodiments, the duration of the effective match window can be set based on tempo and / or on determining a performance skill level based on a predetermined skill level of the user and / or a predetermined difficulty level of the musical score. In some such embodiments, the comparison in stage 520 can identify matching events as HP events that have been determined to temporally correspond to one of a plurality of SNR events by: searching for HP events and determining whether any of the plurality of HP events have a relevant HP onset time within a valid match window for the SNR event; identifying the SNR event as a skipped SNR event in response to determining that none of the plurality of HP events have a relevant HP onset time within a valid match window for the SNR event; identifying the SNR event as an executed SNR event in response to determining that a particular HP event among the plurality of HP events has a relevant HP onset time within a valid match window for the SNR event, such that the executed SNR event has an executed onset time corresponding to the HP onset time of the particular HP event and an executed offset time corresponding to the HP offset time of the particular HP event, and associating the SNR event with the particular HP event as a matching event among a plurality of matching events that have been determined to temporally correspond to the SNR event.

[0083]

[0088] In stage 524, embodiments may display visual feedback via an output device interface to graphically map evaluation data of multiple SNR events to the musical notation representation of the multiple SNR events on a musical score. In some embodiments, the display step in stage 524 includes, for each executed SNR event, a step of graphically mapping the executed onset time and executed offset time of the executed SNR event to the musical notation representation of the SNR event on the musical score. In some embodiments, the display step in stage 524 includes, for each executed SNR event, a step of first graphically mapping the executed onset time and executed offset time of the executed SNR event to the musical notation representation of the SNR event on the musical score, and a step of second graphically mapping the associated SNR onset time and associated SNR offset time of the executed SNR event to the musical notation representation of the executed SNR event on the musical score, wherein the first graphical mapping step is visually distinguishable from the second graphical mapping step. In some embodiments, the display step in stage 524 includes a step of graphically mapping, for each skipped SNR event, the associated SNR onset time and associated SNR offset time of the SNR event to the musical notation representation of the skipped SNR event on the musical score. In some embodiments, the step of obtaining the associated evaluation criterion for each SNR event in stage 516 includes a step of calculating a valid offset window for the associated SNR offset time, and the display step in stage 524 includes a step of graphically mapping, for each executed SNR event, the complete performance instruction to the musical notation representation of the executed SNR event on the musical score, only in response to a determination that the executed offset falls within a valid offset window.

[0084]

[0089] The methods, systems, and devices described above are examples. In various configurations, various steps and components may be omitted, replaced, or added as needed. For example, in an alternative configuration, the method may be performed in a different order than described, and / or various stages may be added, omitted, and / or combined. Also, features described for a particular configuration may be combined in various other configurations. Different aspects and elements of a configuration may be combined in similar ways. Furthermore, because technology is evolving, many of the elements are examples and do not limit the scope of this disclosure or the claims.

[0085]

[0090] Certain details are described in order to provide a complete understanding of the configuration examples (including implementation forms). However, the configuration may be carried out without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary details to avoid obscuring the configuration. This description provides only configuration examples and does not limit the claims, applicability, or configuration. Rather, the foregoing description of the configuration will provide a practical description for carrying out the described techniques for those skilled in the art. Various modifications can be made to the function and arrangement of the elements without departing from the spirit or scope of this disclosure.

[0086]

[0091] Furthermore, the configuration may be described as a process shown as a flow chart or block diagram. Each may describe its operation as a sequential process, although many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. The process may have additional steps not included in the diagram. Furthermore, examples of methods may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a non-temporary computer-readable medium such as a storage medium. The processor may perform the tasks described.

[0087]

[0092] While several exemplary configurations have been described, various modifications, alternative configurations, and equivalents can be used without departing from the spirit of this disclosure. For example, the elements described above may be components of a larger system, and other rules may take precedence over the application of the present invention, or the application may be modified. Also, several steps may be taken before, during, or after considering the elements described above.

Claims

1. A method for tactile-based evaluation of rhythmic performance with reference to a musical score, wherein the method is: A step of outputting an audiovisual representation of real-time progress through a musical score at a certain tempo via the output device interface of a computing device, wherein the musical score includes a musical notation representation of each of a sequence of score-noted rhythm (SNR) events, and each SNR event has a stored logical definition including an associated SNR onset time and an associated SNR offset time, The step of obtaining a plurality of haptic performance (HP) events during the output step, corresponding to user interaction with the haptic device interface of the computing device representing the user's rhythmic performance of the musical score at the tempo, wherein each HP event has an associated HP onset time and an associated HP offset time. The following steps: For each SNR event in a series of SNR events, the steps include obtaining relevant evaluation criteria for the SNR event based on the relevant SNR onset time and the relevant SNR offset time, and A step of comparing the associated evaluation criteria for each SNR event with the associated HP onset time and associated HP offset time for each of at least some of the HP events among the plurality of HP events, wherein the comparison identifies the plurality of matching events as belonging to the HP events that are determined to be temporally corresponding to one of the set of SNR events, and the rhythmic compatibility between each of the plurality of matching events and the temporally corresponding one of the set of SNR events is evaluated. The steps include generating evaluation data for the series of SNR events by a rhythm evaluator subsystem that communicates with the output device interface and the haptic device interface, The steps include: displaying visual feedback via the output device interface to graphically map the evaluation data of the series of SNR events to the musical notation representation of the series of SNR events on the musical score; Methods that include...

2. The output device interface of the computing device is a touch screen interface. The step of outputting the audiovisual representation includes the step of outputting a graphics performance interface via the touchscreen interface, thereby, The first part of the graphics performance interface displays the music score having the musical notation representation of the series of SNR events, The second portion of the graphics performance interface is designated for haptic input, thereby the step of acquiring the plurality of HP events corresponds only to the user interaction with the haptic device interface within the second portion of the graphics performance interface. The method according to claim 1.

3. The step of outputting the aforementioned audiovisual representation is, A step of outputting a graphic representation of the boundary of the second portion of the graphics performance interface where the user interaction with the haptic device interface is recognized as an HP event, wherein the second portion of the graphics performance interface does not completely overlap with the first portion of the graphics performance interface. The method according to claim 2, further comprising:

4. The second part of the graphics performance interface is A first region graphically designated by the haptic device interface for left-hand user interaction, wherein the left-hand user interaction is recognized as a left-hand HP event, and a first region, A second region graphically designated by the haptic device interface for right-hand user interaction, wherein the right-hand user interaction is recognized as a right-hand HP event, and the second region does not completely overlap with the first region. The method according to claim 2, including the method described in claim 2.

5. The output step establishes the performance time standard such that each HP onset time and each HP offset time corresponds to the respective time values ​​of the performance time standard. The step of generating the evaluation data for the series of SNR events includes, for each SNR event, mapping the associated SNR onset time and the associated SNR offset time to the performance time criterion, wherein the associated evaluation criterion for each SNR event is defined relative to the performance time criterion; and comparing the associated evaluation criterion for each SNR event being executed with the performance time criterion. The method according to claim 1.

6. The step of obtaining the relevant evaluation criteria for each SNR event includes the step of calculating a match window that is valid for the SNR onset time, The aforementioned comparison step is, A step of searching the plurality of HP events to determine whether any of the plurality of HP events has a relevant HP onset time within the valid match window of the SNR event, In response to determining that none of the aforementioned multiple HP events have a corresponding HP onset time within the valid match window of the SNR event, the SNR event is identified as a skipped SNR event. In response to determining that a specific HP event among the plurality of HP events has a relevant HP onset time within the valid match window of the SNR event, the steps include identifying the executed SNR event as an executed SNR event such that the executed SNR event has an executed onset time corresponding to the HP onset time of the specific HP event and an executed offset time corresponding to the HP offset time of the specific HP event, and associating the SNR event with the specific HP event as the matching event among the plurality of matching events determined to be temporally corresponding to the SNR event, This identifies the multiple matching events as belonging to the HP event, each of which is determined to temporally correspond to one of the series of SNR events. The method according to claim 1.

7. The aforementioned comparison step is, Steps to set the duration of the effective match window based on the tempo. The method according to claim 6, further comprising:

8. The step of generating the evaluation data includes a step of determining the performance skill level based on a predetermined skill level of the user and / or a predetermined difficulty level of the music score, The comparison step further includes setting the duration of the effective match window based on the performance skill level. The method according to claim 6.

9. The step of displaying the visual feedback in order to graphically map the evaluation data includes, for each executed SNR event, the step of graphically mapping the executed onset time and executed offset time of the executed SNR event to the musical notation representation of the SNR event on the musical score. The method according to claim 6.

10. The step of displaying the visual feedback in order to graphically map the evaluation data is performed for each SNR event that has been performed. The steps include first graphically mapping the executed onset time and executed offset time of the executed SNR event to the musical notation representation of the SNR event on the musical score, The steps include: secondly graphically mapping the associated SNR onset time and associated SNR offset time of the executed SNR event to the musical notation representation of the executed SNR event on the musical score; Includes, The first graphical mapping step is visually distinguishable from the second graphical mapping step. The method according to claim 6.

11. The step of displaying the visual feedback to graphically map the evaluation data includes, for each skipped SNR event, the step of graphically mapping the associated SNR onset time and associated SNR offset time of the SNR event to the musical notation representation of the skipped SNR event on the musical score. The method according to claim 6.

12. The step of obtaining the relevant evaluation criteria for each SNR event further includes the step of calculating a valid offset window for the relevant SNR offset time, The step of displaying the visual feedback includes, for each executed SNR event, graphically mapping the complete performance instruction to the musical notation representation of the executed SNR event on the musical score, only in response to a determination that the executed offset falls within the valid offset window. The method according to claim 6.

13. The output step includes a step of displaying the musical notation representation of the sequence of SNR events on a grand staff such that the sequence of SNR events includes right-hand SNR events and left-hand SNR events. The step of acquiring the plurality of HP events includes the steps of acquiring a first portion of the HP events as a right-hand HP event from a designated right-hand portion of the haptic device interface, and acquiring a second portion of the HP events as a left-hand HP event from a designated left-hand portion of the haptic device interface. The method according to claim 1.

14. The step of generating the evaluation data for the series of SNR events is, The steps include comparing the associated evaluation criteria for each of the right-hand SNR events with the associated HP onset time and associated HP offset time for each of the right-hand HP events only, A step of comparing the associated evaluation criteria for each of the right-hand SNR events with the associated HP onset time and associated HP offset time for each of the left-hand HP events only. The method according to claim 13.

15. The step of outputting the audiovisual representation of the real-time progress through the musical score at the aforementioned tempo, The steps include: displaying the music score via the output device interface; The steps include: playing audio corresponding to at least each of the sequences of SNR events in order to audibly represent the real-time progress through the musical score at the aforementioned tempo; During playback, a graphic position indicator is overlaid on the music score, dynamically updating to show which of the sequence of SNR events corresponds to the currently playing audio, thereby visually representing the real-time progress through the music score at the tempo, in steps and The method according to claim 1, including the method described in claim 1.

16. The step of outputting the audiovisual representation of the real-time progress through the musical score at the aforementioned tempo, During playback, when each of the multiple HP events is acquired, the graphical representation of the associated HP onset time and associated HP offset time for each of the multiple HP events is overlaid and displayed on the music score. The method according to claim 15, further comprising:

17. The step of outputting the audiovisual representation of the real-time progress through the musical score at the aforementioned tempo, A step to play the rhythm track at the aforementioned tempo, synchronized with the playback of the audio. The method according to claim 15, further comprising:

18. Each of the first subset of the series of SNR events corresponds to a note event, and the musical notation representation of each note event indicates one or more notes of one or more pitches to be held over a notated duration that begins at the SNR onset time of the corresponding SNR event and ends at the SNR offset time of the corresponding SNR event. Each of the second subset of the series of SNR events corresponds to a rest event, and the musical notation for each rest event indicates silence to be maintained over the notated duration, beginning at the SNR onset time of the corresponding SNR event and ending at the SNR offset time of the corresponding SNR event. The method according to claim 1.

19. The rhythm evaluator subsystem is implemented by one or more processors of the computing device. The method according to claim 1.

20. The rhythm evaluator subsystem is implemented by a server that communicates with the computing device via a communication network. The method according to claim 1.

21. The user interaction with the haptic device interface includes at least one of touching a trackpad, gripping a force sensor, clicking a mouse button, pressing a keyboard key, or gripping a haptic glove. The method according to claim 1.

22. The acquired user interaction is not accompanied by an acoustic signal. The method according to claim 1.

Images