Musical performance evaluation system, musical performance evaluation method, and non-transitory computer-readable medium storing musical performance evaluation program

Abstract

To provide a musical performance evaluation system having a score independent conspicuous error detection function.SOLUTION: A musical performance evaluation system basically includes an audio input, a notification device, and at least one processor. The audio input is configured to input a musical performance. The notification device outputs an evaluation of the musical performance received via the audio input. The at least one processor executes a musical performance evaluation program to identify errors in the musical performance based on reference data of segments of musical performances containing errors, classify the errors in the musical performance as either a conspicuous error or an inconspicuous error, and instruct the notification device to output the evaluation of the musical performance identifying a presence of conspicuous errors different from inconspicuous errors.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a performance evaluation system, a performance evaluation method, and a non-transitory computer-readable recording medium storing a performance evaluation program. More specifically, the present disclosure relates to providing feedback regarding the presence of errors (misplays or mistakes) in a musical performance. [Background technology]

[0002] A commonly held view in automated music performance analysis (MPA) research is that any deviation in a musical performance from its underlying score can be considered a performance error. However, research in music pedagogy suggests that some deviations in musical performance are more obvious to listeners than others. For example, voicing a chord differently than written in the score might be overlooked, but missing a note in a characteristic motif or playing a note that clashes with the underlying harmony would be noticeable. Thus, some errors in musical performance are more noticeable than others.

[0003] In recent years, the music industry has developed a variety of computer-assisted devices intended to teach and / or improve a student's ability to play a musical instrument, such as the piano. For example, a variety of musical instrument teaching software and apps have been developed to teach and / or improve a student's ability to play a musical instrument. These teaching software and apps do not distinguish between types of errors in musical performance. Rather, all errors in musical performance are treated the same. Summary of the Invention [Problem to be solved by the invention]

[0004] Music education software that provides analysis based only on strict note-level rhythm and pitch correctness has been problematic because it can lead users to focus too much on playing correctly (almost robotically) in order to get the highest score. It has been found that there are many considerations to make in order to design useful music education software that automatically evaluates performance. For example, it has been found that beginners and intermediate players need different feedback on their performance than advanced players.

[0005] The present disclosure is primarily directed to a performance evaluation system that provides performance feedback to beginner and intermediate students. In particular, in a musical performance, some mistakes or errors may be noticeable to a listener, while others may go unnoticed. How noticeable a mistake is depends on factors such as the contextual appropriateness of the mistake and the listener's familiarity with the musical performance being performed.

[0006] A noticeable error or mistake is considered to be an error or mistake that is obvious to a listener, regardless of the listener's level of familiarity with the musical performance being performed. More specifically, a noticeable error is considered to be a performance mistake that would be detectable by a majority of listeners with formal musical training, regardless of their level of familiarity with the musical notation underlying the piece being performed.

[0007] Of course, noticeable errors depend on the listener's knowledge of the piece and the performer's proficiency. Moreover, noticeable errors and expression are two sides of the same coin. For example, hitting an adjacent key can be conveyed as an expressive ornament or as a noticeable error. This suggests that the detection of noticeable errors should be conditioned by the style, the listener's level, and the performer's proficiency.

[0008] One aspect of the present disclosure is to infer, for a given sequence of music, a time sequence of binary labels for evaluating a musical performance by indicating the presence of noticeable errors at given times.

[0009] Another aspect of the present disclosure is to provide a performance assessment system having sheet music independent noticeable error detection capabilities to assist students from beginner to intermediate level by assessing their musical performances. [Means for solving the problem]

[0010] According to one aspect of the present disclosure, there is provided a performance evaluation system basically comprising: an audio input; a notification device; and at least one processor. The audio input is configured to input a performance. The notification device is configured to output an evaluation of the musical performance. The at least one processor is operatively coupled to the computer readable storage medium and the notification device. The at least one processor is configured to execute a performance evaluation program to instruct the notification device to identify errors in the musical performance based on the reference data, classify the errors in the musical performance as either noticeable errors or non-noticeable errors, and output an evaluation of the musical performance distinguishing between the noticeable errors and the non-noticeable errors.

[0011] In another aspect of the present disclosure, a computer-implemented method for evaluating a musical performance is provided, the method including obtaining a musical performance by a user, identifying, using at least one processor, errors in the musical performance based on the reference data, classifying, using the at least one processor, the errors in the musical performance into either noticeable errors or non-noticeable errors, and instructing a notification device to output an evaluation of the musical performance distinguishing between the noticeable errors and the non-noticeable errors.

[0012] In another aspect of the present disclosure, a non-transitory computer-readable recording medium having stored thereon a performance evaluation program is provided, which, when executed by a computing device, causes the computing device to obtain a musical performance by a user, identify errors in the musical performance based on the reference data using at least one processor of the computing device, classify the errors in the musical performance into either noticeable errors or inconspicuous errors using at least one processor of the computing device, and instruct a notification device to output an evaluation of the musical performance by distinguishing between the noticeable errors and the inconspicuous errors.

[0013] Furthermore, other objects, features, aspects and advantages of the disclosed performance evaluation system will become apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a simplified perspective view of a performance evaluation system according to an embodiment in which the musical instrument is an electronic keyboard. [Diagram 2] FIG. 1 is a block diagram of a performance evaluation system according to an embodiment. [Diagram 3] FIG. 1 is an illustration of a Temporal Convolutional Network (TCN) having a feature extraction backbone and a classifier head that receives and evaluates a musical performance and identifies noticeable errors in the musical performance. [Figure 4] This is a simplified diagram of a performance evaluation system. [Diagram 5] 1 is a flow chart illustrating an evaluation process performed by a computing device of the performance evaluation system. [Figure 6] 1 is an illustration of an autoencoder used to train a temporal convolutional network (TCN) feature extractor using reference data of musical performances. [Figure 7]FIG. 1 illustrates a piano roll created based on a performance and the identification of errors in the performance using a performance evaluation system. [Figure 8] 8 is an illustration of a section (segment) of music corresponding to error "A" in FIG. 7, which has been identified by a performance assessment system as containing at least one noticeable error. [Figure 9] 8 is an illustration of a section (segment) of music corresponding to error "B" in FIG. 7, which was identified by the performance assessment system as containing at least one subtle error. [Figure 10] FIG. 8 is an illustration of a section (segment) of music corresponding to error “C” in FIG. 7, which was identified by the performance assessment system as containing at least one subtle error. [Figure 11] 8 is an illustration of a section (segment) of music corresponding to error "D" in FIG. 7, which was identified by the performance assessment system as containing at least one noticeable error. [Figure 12] FIG. 8 is an illustration of a section (segment) of music corresponding to error “E” in FIG. 7, which has been identified by the performance assessment system as containing at least one noticeable error. [Figure 13] FIG. 1 illustrates a recording screen of a digital audio workstation (DAW) used to record a musical performance and create a piano roll of the musical performance. [Figure 14] FIG. 1 illustrates a recording review screen of a digital audio workstation in which a recorded musical performance has been evaluated using a performance evaluation system to identify the presence of noticeable errors. [Figure 15] FIG. 1 illustrates an output screen of a digital audio workstation in which the presence of both noticeable and non-noticeable errors is identified in a recorded musical performance. [Figure 16] FIG. 1 shows an output screen of a digital audio workstation in which only noticeable errors are identified in a recorded musical performance. [Figure 17]FIG. 1 illustrates an output screen of a digital audio workstation showing a recorded musical performance in which the presence of both noticeable and non-noticeable errors has been identified in the recorded musical performance, but the noticeable errors are emphasized more than the non-noticeable errors. [Figure 18] FIG. 13 illustrates a digital audio workstation output screen showing a virtual listener avatar's evaluation of a section of a musical performance that is free of noticeable errors. [Figure 19] FIG. 13 illustrates an output screen of a digital audio workstation showing an assessment by a virtual listener avatar upon identifying the presence of at least one noticeable error in a section of a musical performance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Next, the embodiments will be described with reference to the drawings. It will be apparent to those skilled in the art of music from this disclosure that the following description of the embodiments is provided for illustrative purposes only, and is not intended to limit the present invention defined by the appended claims and their equivalents.

[0016] Referring first to Fig. 1, a performance evaluation system 10 including a performance evaluation program according to an exemplary embodiment is shown. In the exemplary embodiment of Fig. 1, the performance evaluation system 10 is used with a digital piano 12 to obtain a musical performance played on the digital piano 12. In other words, in the exemplary embodiment of Fig. 1, the digital piano 12 is an example of a musical instrument used with the performance evaluation system 10. As will be apparent from this disclosure, the performance evaluation system 10 can be used with other types of musical instruments and other devices on which music can be played, such as laptops, personal computers, tablets, etc.

[0017] Essentially, as will be described in more detail below, the performance evaluation system 10 shown in Figure 1 is configured to provide feedback to a user or student on a musical performance played on an instrument such as a digital piano 12. The performance evaluation system 10 is particularly useful for beginners and intermediate players. The performance evaluation system 10 is configured to focus on obvious errors or mistakes made by a performer during a performance.

[0018] The performance evaluation system 10 is preferably configured to evaluate a musical performance even if the musical performance is not preloaded with sheet music for the musical performance into the performance evaluation system 10. However, as will be explained below, the performance evaluation system 10 is not limited to this preferred methodology. Evaluation of a musical performance may also be accomplished by comparing the musical performance to sheet music for the musical performance that is preloaded into the performance evaluation system 10.

[0019] 1, the performance evaluation system 10 also includes a computing device 14. In the exemplary embodiment of FIG. 1, the computing device 14 is a laptop computer having a display 16, a keyboard 18, and a mouse 20.

[0020] The display 16 includes, for example, a liquid crystal display, and displays the results of the evaluation process performed by the performance evaluation system 10. The display 16 is an example of a notification device of the performance evaluation system 10. The keyboard 18 and the mouse 20 are examples of input devices of the computing device 14. The keyboard 18 and the mouse 20 are operated by a user to make a predetermined selection or designation.

[0021] Although computing device 14 is illustrated as a laptop computer, computing device 14 is not limited to a laptop computer as illustrated in Figure 1. For example, computing device 14 may include a personal computer, a tablet computer, a handheld computer, a personal digital assistant (PDA), a smartphone, etc.

[0022] Also, a touch screen may be substituted for the display 16, keyboard 18, and mouse 20. In any event, the various portions of the computing device 14 may be integrated into a single unit or may be separated into two or more separate units as is well known in the computing arts.

[0023] Here, in the exemplary embodiment of FIG. 1, the performance evaluation system 10 includes an external speaker 22. The speaker 22 is another example of a notification device of the performance evaluation system 10. Alternatively, an internal speaker can be used to output a notification indicating the result of the evaluation process performed by the performance evaluation system 10. Of course, the notification device of the performance evaluation system 10 is not limited to the example shown in the figure. The notification device of the performance evaluation system 10 may be any device that outputs at least one of a visual notification, a tactile notification, and an audio notification to notify the user of the result of the evaluation process performed by the performance evaluation system 10.

[0024] Here, in the exemplary embodiment of FIG. 1, the performance evaluation system 10 is configured to acquire a performance of a musical instrument (e.g., a digital piano 12) played by a user. For example, the performance evaluation system 10 includes a communication cable 24 that connects the computing device 14 to the digital piano 12 or other musical instrument. Alternatively, for example, the performance evaluation system 10 includes a microphone 26 that is connected to the computing device 14 to acquire a performance on the digital piano 12 or other musical instrument. The communication cable 24 (digital connection) and the microphone 26 are examples of an audio input (input device) configured to input a performance played on a musical instrument to the computing device 14. Of course, the audio input can be formed by other sound processing devices as needed and / or desired. In other words, the performance evaluation system 10 includes an audio input configured to input a performance played on a musical instrument to the computing device 14. Also, in the exemplary embodiment of FIG. 1, the performance evaluation system 10 can communicate with a remote server 28 (e.g., a network server or a cloud server). The remote server 28 includes at least one processor and a computer-readable storage device (i.e., a computer memory).

[0025] Referring to Fig. 2, a block diagram is shown illustrating an example of an overall hardware configuration of the performance evaluation system 10 of Fig. 1. As seen in Fig. 2, the computing device 14 includes at least one processor 30 and a computer-readable storage device 32 (i.e., computer memory). The processor 30 is configured to execute programs stored on a remote server 28 (e.g., a network server or a cloud server) or the computer-readable storage device 32. The computing device 14 may include one or more types of processors, such as a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc., instead of or in addition to the processor 30.

[0026] As used herein, the term "processor" refers to hardware that executes a software program and does not include humans. As used herein, the term "computer-readable storage medium" refers to any non-transient computer storage device and does not include a transitory propagating signal. For example, computer-readable storage device 32 includes storage media such as hard disks, optical disks, magnetic disks, memory cards, etc., and stores a learning model as described below.

[0027] Preferably, the computer-readable storage device 32 includes a random access memory (RAM) and a read only memory (ROM). The RAM of the computer-readable storage device 32 is a volatile memory that is used as a work area for the processor 30 and temporarily stores various data. The ROM is a non-volatile memory that stores various support programs.

[0028] The processor 30 and the computer readable storage device 32 are formed of one or more semiconductor chips mounted on a circuit board. The processor 30 is electrically connected to the computer readable storage device 32. The processor 30 is also operatively coupled to the display 16 (i.e., notification device), the keyboard 18, the mouse 20, the speaker 22 (i.e., notification device), the communication cable 24 (i.e., audio input), and the microphone 26 (i.e., audio input).

[0029] The processor 30 realizes various functions by executing various functions stored in a computer-readable storage device 32. In the illustrated example of Fig. 2, the execution of programs by the processor 30 includes the execution of an application program implementing a digital audio workstation (DAW) 34 and the execution of a performance evaluation program 36. The processor 30 also executes an operating system and other subroutine programs that are executed in response to a user's instruction.

[0030] The performance evaluation program 36 executed by the processor 30 can be recorded on a non-transitory computer-readable recording medium such as a magnetic recording medium (magnetic tape, magnetic disk, etc.), an optical recording medium (optical disk, etc.), a magneto-optical recording medium, a semiconductor memory, etc. The non-transitory computer-readable recording medium can be provided to the computing device 14 and stored in the computer-readable storage device 32 of the computing device 14.

[0031] The digital audio workstation 34 is a software program stored on the computer readable storage device 32. Alternatively, the digital audio workstation 34 may be stored on a computer readable storage device of a remote server 28 (e.g., a network server or a cloud server).

[0032] The digital audio workstation 34 is typically configured to compose, produce, record, mix, and edit audio and MIDI data. The digital audio workstation 34 may be commercially available or proprietary software capable of recording a musical performance played on at least the instrument being evaluated. Because digital audio workstations are well known, the digital audio workstation 34 will not be described in further detail herein. In any case, the digital audio workstation 34 is configured to record the musical performance and provide MIDI note events of the musical performance as input to the at least one processor 30.

[0033] Preferably, the MIDI note events of the musical performance include a start time, an end time, a pitch, and a velocity of each note event of the musical performance, where the digital audio workstation 34 creates at least one piano roll of the musical performance as an input to the at least one processor 30. Preferably, the at least one piano roll of the musical performance includes a first piano roll that provides data regarding the onset of notes in the musical performance, and a second piano roll that provides data regarding sustained portions of key presses in the musical performance, as described below.

[0034] Next, the performance evaluation program 36 will be described in more detail with reference to Figures 3 to 5. The performance evaluation program 36 is a software program stored in the computer readable storage device 32. Alternatively, the performance evaluation program 36 can be stored in a computer readable storage device of a remote server 28 (e.g., a network server or a cloud server). At least one processor 30 is configured to execute the performance evaluation program 36 to classify performance errors using the learning model of the performance evaluation program 36.

[0035] The performance evaluation program 36 basically includes a Temporal Convolutional Network (TCN) 40 and a classifier head 42. The classifier head 42 is provided after the temporal convolutional network 40. Therefore, the performance evaluation program 36 is a TCN-based network. The temporal convolutional network 40 and the classifier head 42 form a machine learning model, and may also be simply called a learning model.

[0036] Here, a temporal convolutional network 40 takes as input a musical performance (e.g., at least one piano roll) and emits binary labels of prominent errors for each predefined time window of the piano roll. Preferably, the binary labels are assigned at the frame level rather than the note level. The temporal convolutional network 40 and the classifier head 42 can also be referred to as a prominent error detector.

[0037] If the performance evaluation system 10 is connected to a network such as the Internet, the learning model may be stored in a remote server 28 (e.g., a network server or a cloud server) instead of the computer-readable storage device 32. As described in more detail below, the performance evaluation program 36 is preferably a score-independent evaluation program. However, as described below, the performance evaluation program 36 is not limited to score-independent evaluation programs.

[0038] Essentially, the performance evaluation system 10 evaluates a performance by comparing the performance to reference data stored in the computer readable storage device 32 or a remote server 28 (e.g., a network server or a cloud server). The reference data is collected to identify errors or mistakes made by the performer in performing the musical performance. In the main embodiment, the reference data does not include a complete score of an error-free musical performance (i.e., a musical score of the musical performance). Rather, in the main embodiment, the reference data consists of a plurality of inaccurate performances having errors, where the errors in the inaccurate performances have been identified.

[0039] For example, if the musical instrument is a piano, the reference data may be obtained by obtaining MIDI data of multiple actual piano performances performed by pianists of various skill levels and / or by creating synthetic data of multiple piano performances with procedurally generated mistakes. In either case, errors or mistakes in the performances of the MIDI data and / or synthetic data are identified and classified. In particular, the reference data is annotated with areas believed to contain one or more noticeable errors. Thus, as used herein, the term "reference data" refers to a set of data representing multiple sections of a performance, including annotations in areas believed to contain one or more noticeable errors.

[0040] Basically, in the performance evaluation system 10, the computer-readable storage device 32 has reference data of the section of the performance including the error. Alternatively, the reference data of the section of the performance including the error can be stored in a remote server 28 (e.g., a network server or a cloud server). The notification device (e.g., the display 16 and / or the speaker 22) outputs the evaluation of the musical performance received via the audio input (e.g., the communication cable 24 and / or the microphone 26).

[0041] The at least one processor 30 executes a performance evaluation program 36 to identify errors in the musical performance based on reference data stored in the computer readable storage device 32, classify the errors in the musical performance as either noticeable errors or non-noticeable errors, and instruct the notification device (e.g., display 16 and / or speaker 22) to output a rating of the musical performance that identifies the presence of noticeable errors as distinct from non-noticeable errors. Preferably, the at least one processor 30 is configured to instruct the notification device (e.g., display 16 and / or speaker 22) to output a rating of the musical performance that identifies only the presence of noticeable errors.

[0042] Next, some examples of reference data that can be used when evaluating a performance using the performance evaluation system 10 will be described. As described above, the reference data can include MIDI data of multiple performances and / or composite data of multiple performances in which procedural errors have been generated. When acquiring MIDI data of an actual piano performance, for example, three qualitatively different piano performance MIDI data can be acquired to identify errors or mistakes made by a performer in playing the piano.

[0043] The first set of piano performance MIDI data may include a number of sight-reading performances by formally musically trained beginner and intermediate adult pianists. The first set of piano performance MIDI data is referred to herein as sight-reading (SR) data. The sight-reading data performances may consist primarily of piano reductions of popular classical pieces arranged for beginner to intermediate levels.

[0044] The second set of piano performance MIDI data may include a number of performances by pianists ranging from high beginner to early advanced. The second set of piano performance MIDI data is referred to herein as performance data (PF). Each performance in the performance data is approximately three minutes long and is collected from a digital piano recording app. Not all of the performances included in the performance data are known, but most are pop or classical pieces, loaded from sheet music or semi-adapted. The demographics of the users of the performance data are unknown, but the performance data suggests a skill level between late beginner and early advanced.

[0045] The third set of piano performance MIDI data may include performances of a predetermined number of Bergmuller's 25 Etudes by an advanced pianist. The third set of piano performance MIDI data is referred to herein as Bergmuller data (BM:Bergmuller). The performances of Bergmuller's 25 Etudes may be recorded twice on a digital piano. The performances of Bergmuller's 25 Etudes are performed by an advanced pianist who has previously performed the etudes. In the Bergmuller data, the pianist briefly practices each etude before recording two takes.

[0046] for example, (1) 103 sight-reading sessions conducted by adult pianists at the beginner to intermediate level who had received formal music education; (2) 245 performances on digital pianos by pianists at the late beginner to early advanced level; (3) A test was conducted on piano performance MIDI data comprising 50 etude performances by an advanced pianist. The data for this test was annotated with regions believed to contain noticeable errors. The test then used a temporal convolutional network 40 to detect noticeable locations of such errors in a piano roll of the performance. The output from the temporal convolutional network 40 for each detected error was then processed using a classifier head 42 to determine the probability that each detected error was a noticeable error. Finally, the piano roll of the performance was displayed and only the noticeable errors in the performance were identified.

[0047] The total duration of the sight reading data is a first predetermined duration. The total duration of the performance data is a second predetermined duration. The total duration of the Bergmuller data is a third predetermined duration. The first, second, and third predetermined durations can be set as needed and / or desired. The sight reading data and performance data are split and used for training, validation, and testing with no overlaps, and the Bergmuller data is reserved for testing only. In each data set, noticeable errors were annotated as noticeable errors. An example annotation procedure is described herein. Alternative annotation procedures can be used as needed or desired.

[0048] In the illustrated example, at least two annotators are preferably used. The first annotator is an experienced classical piano teacher. The second annotator is trained in music production and is also an intermediate level pianist. The first annotator labels the sight reading data and the Bergmuller data, while the second annotator labels the performance data. In each case, the first and second annotators are to indicate whether they know the piece being played (yes / no). In addition, for the sight reading data and the performance data, the annotators are instructed to annotate obvious performance errors that can be seen without checking the sheet music (hereafter referred to as conspicuous errors), and it is up to the first and second annotators to decide what constitutes a conspicuous error. The annotation of the MIDI data can be done in music production software such as Cubase. The first and second annotators add annotations to MIDI note 0 covering the span of a time window that they determine to be relevant to the error.

[0049] The Bergmüller data were treated differently because they were performed from known score data. First, deviations in the performance in the score were automatically annotated using a score alignment system. A first annotator then manually reviewed the labels by listening to the performance while looking at the corresponding score, adding deviations from the score or removing those that did not reflect errors. The first annotator simultaneously manually labeled each error as either noticeable or not. Although the ratio of annotation area to total performance time may be small for the Bergmüller data, the annotation approach allows us to examine the relationship between the set of errors (presumably all errors) obtained by comparing them to the score and the noticeable errors.

[0050] In this methodology, some types of errors are more consistently labeled than others. More common errors include the insertion or deletion of notes that do not fit the musical context, abrupt pauses, and erratic rhythms resulting from hesitations during playing. Common errors are annotated with reasonable consistency in terms of label location and span. Common errors are relatively short, localized errors after which the player regains the flow of the performance, and are annotated with reasonable consistency in terms of label location and span. More complex deviations, however, are labeled ambiguously. For example, a player may "sneak" a practice before resuming the flow of the piece after an error. In such instances, where the short phrase being practiced sounds out of context but is itself coherent, it remains an open question as to where the label should be placed, whether it should be labeled as continuous or intermittent.

[0051] Additionally, while there are noticeable errors in the data that are not annotated, there is an inherent ambiguity in how to assess "bad but acceptable" versus "incorrect performance." If an area contrasts with the annotator's expectations of the music given the performer's style of playing, it will be annotated. This raises the possibility that annotators may be adjusting what counts as an error based on individual performances. Silence is one major source of ambiguity, since silence between correct segments is not annotated regardless of its length, whereas silence within or around mistakes is often labeled as an error.

[0052] Because annotating real performances is time-consuming and difficult to obtain, two pre-training methods are proposed to overcome the lack of data. The first pre-training method trains part of the model as an autoencoder. The second pre-training method uses synthetic data with procedurally generated errors. Experiments show that TCN has an F-measure of 78% without pre-training on sight-reading data. However, the proposed pre-training method improves the F-measure on both performance and Bergmuller data to a degree that approaches the salient error labels of human annotators.

[0053] First, a pre-training method using an autoencoder 44 will be described. The autoencoder 44 is used to train a feature extractor in an unsupervised manner using a collection of piano performances. The autoencoder 44 is composed of an encoder 46 and a decoder 48. Specifically, in the illustrated example, the encoder 46 uses feature extraction from a temporal convolutional network 40, and the decoder 48 uses a temporal convolutional network in which a one-dimensional convolution is transposed instead of a one-dimensional convolution.

[0054] Encoder 46 is provided with MIDI data of a musical performance with unknown performance quality. Encoder 46 compresses the input (e.g., piano roll MIDI data) and decoder 48 attempts to recreate the input (e.g., piano roll MIDI data) from the compressed version provided by encoder 46. After training, encoder 46 is saved and decoder 48 is discarded.

[0055] The encoder 46 can be used as a data preparation technique to perform feature extraction on raw data that can be used to train the machine learning model of the temporal convolutional network 40. The encoder 46 learns how to interpret the input and compress the input into an internal representation defined by the bottleneck layer. The decoder 48 receives the output of the encoder 46 (the bottleneck layer) and attempts to recreate the input. Once the autoencoder 44 is trained, the decoder 48 is discarded and only the encoder 46 is kept and used to compress the input into a vector that is output by the bottleneck layer.

[0056] In this way, the space φ is pre-trained to model the piano performance space within a given receptive field of the temporal convolutional network 40. This method is useful when a large data set of performances of unknown performance quality is available. This method is useful when a large data set of performances of unknown performance quality is available.

[0057] Next, a pre-training method using synthetic data containing procedurally generated errors is described. The machine learning model is pre-trained with performance data in which performance errors are simulated and error labels are inserted to match the data format that should be obtained by human annotators listening to and annotating musical performances. Specifically, a set of error-free performances is systematically adjusted and note events are corrected based on the performance errors made by novice adult pianists. For example, the error-free performances can be commercially available MIDI piano data that mainly includes jazz and classical piano MIDI performances.

[0058] For each note event with probability pc, the note is modified in one of the following ways: 1. A note is omitted with probability po. 2. To simulate key mis-presses, the same note is replaced by a note transposed by n semitones with probability pr. 3. A note transposed by n semitones is inserted with probability pi. 4. Performance is temporarily halted for a small amount of time uniformly distributed between 0.3 and 0.8 seconds, with probability pp. 5. The last played note is repeated with probability ppr. 6. The performance is temporarily halted with probability ps for a duration uniformly distributed between 2 seconds and 4 seconds.

[0059] In this embodiment, the probabilities are set as follows. pc=5% po=10% pi=39% pr=39% ps=2% pp=10%

[0060] For note substitutions and insertions, n is chosen so that n=1,2 is chosen 22% of the time, and n=4,6 is chosen 2% of the time. This method is useful when a large number of performances are available that are known to be relatively error-free. Furthermore, since not all synthesis errors are noticeable, this method can also be used to extend the data.

[0061] As mentioned above and illustrated in FIG. 3, the conspicuous error detector is formed as a simple temporal convolutional network (TCN) consisting of a feature extraction backbone followed by a classifier head. In the feature extraction backbone of the temporal convolutional network 40, for a given piano roll, the feature extraction backbone selects features φ∈ RD×TIn this example, D=256. That is, the temporal convolutional network (TCN) 40 is preferably realized as a 5-layer acausal TCN with a dilation of [1, 2, 4, 8, 16]. The temporal convolutional network 40 has a channel size of 256, a kernel size of 3, uses ELU nonlinear arithmetic, and has residual connections in all layers.

[0062] As mentioned above and illustrated in Figure 3, the classifier head 42 is a neural network consisting of three layers of 1x1 convolutions with output channel sizes [256,64,1], with residual connections and an ELU nonlinear operation followed by a sigmoid function. Thus, the classifier head 42 computes the probability of noticeable error for a given feature φ.

[0063] In the example in Figure 3, the machine learning model is trained on a 10-fold gradient using RAdam so that the cross-entropy between the probability of noticeable errors and the posterior distribution calculated from the ground-truth labels is minimized. -3 The models were trained with a learning rate of 100%. Reference data was added by randomly transposing the entire MIDI file in the training data. In addition, in the calculation of the cross-entropy loss, the ground truth labels were smoothed to take into account the annotation discrepancies for the start and end times of the notable error intervals. Furthermore, since annotations of notable errors are difficult to obtain, the machine learning model is preferably pre-trained using one of the two pre-training methods mentioned above.

[0064] In the example of FIG. 3, the machine learning model temporal convolutional network 40 is trained using one of the training methods illustrated below.

[0065] 1. Baseline Method The machine learning model, a temporal convolutional network 40, is trained using sight reading data and performance data.

[0066] 2. SYNTH method The machine learning model temporal convolutional network 40 is trained with the same subset of synthetic data used in training and validation as in the baseline method.

[0067] 3. SYNTH (FT) method The machine learning model's temporal convolutional network 40 is fine-tuned using sight reading and performance data after a pre-training run on synthetic data, simulating the situation where a new annotated dataset becomes available after the machine learning model has been trained on synthetic data alone.

[0068] 4.AE method The machine learning model, the temporal convolutional network 40, is pre-trained by the autoencoder 44 using, for example, about 100,000 MIDI performances played by various users as a pre-training step of the backbone TCN. The set of MIDI performances does not include sight reading data, performance data, or Bergmuller data.

[0069] 5.AE+SYNTH method The machine learning model temporal convolutional network 40 is pre-trained by the autoencoder 44 using, for example, about 100,000 MIDI performances played by various users as a pre-training step of the backbone TCN. The machine learning model temporal convolutional network 40 is then fine-tuned using the sight reading data, performance data, and synthesis data.

[0070] We evaluated each of these learning methods and created trained machine learning models. The trained machine learning models were then validated against the sight reading and performance data, and evaluated on a test split of the sight reading and performance data, and on the entire Bergmuller data.

[0071] In these tests, the estimated and ground truth annotations were treated as note events occurring at predefined pitches, and transcription precision / recall / F-measure were evaluated using mir_eval as the metric.

[0072] In computing the transcription metrics, the margin of error for note start and end was set to 2 seconds. Furthermore, based on the validation set, the edges of estimated intervals were padded by 0.2 seconds and overlapping intervals were merged. Test results of the trained machine learning models are shown in Tables 1, 2 and 3.

[0073] Sight Reading (SR) Data [Table 1]

[0074] Performance (PF) data [Table 2]

[0075] Bergmuller (BM) Data [Table 3]

[0076] For the Performance and Bergmuller datasets, some improvements are seen with the augmentation strategy. The two proposed training methods, namely the use of synthetic data and autoencoders44, also bring improvements. In general, it is suggested that both training methods tend to improve recall and provide similar qualitative improvements, and one or the other can be adopted depending on the available data.

[0077] As shown in Tables 2 and 3, despite the augmentation strategy, the F-measure precision of the performance data and the Bergmuller data is relatively low even when the ambiguity of conspicuous errors is taken into account. In particular, the F-measure precision of the performance data and the Bergmuller data is relatively low compared to the F-measure precision of the sight-reading data, even when the ambiguity of conspicuous errors is taken into account. As shown in Tables 2 and 3, inference is difficult for the performance data and the Bergmuller data, as can be seen from the difference in F-measures between the sight-reading data set.

[0078] As another example, the F-measure validation value of the trained machine learning model on the synthetic dataset is about 60%. This suggests that the machine learning model is moderately capable of pinpointing the correct label when the labels are easy to classify or when the labels are generated systematically, albeit probabilistically. However, at the same time, the trained machine learning model falls short of the F-measure target accuracy score of 43%, as shown by the best-performing F-measure of 38% on the performance dataset.

[0079] The machine learning model trained on the sight-reading data performed best, likely because most mistakes are highly visible in the sight-reading situation, especially when compared to the performance data and the Bergmuller data (which are mostly beginner to intermediate players with occasional mistakes).

[0080] The performance of trained machine learning models tends to decrease as the number of pre-training steps increases, which is presumably because the pre-training data mostly contains the same type of data as the performance dataset, increasing the gap between the training data and the test data.

[0081] These results suggest that in the context of sightreading, instead of pre-training or augmenting a dataset with a typical amateur performance containing some noticeable errors, it is sufficient to simply train with a dataset that contains only the same set of data.

[0082] The methods used in the learning models above tend to identify repetitions, pauses, hesitations, and the insertion of notes that occur at closely spaced intervals as performance errors, but at the same time, the very same characteristics arising from musical expression and composition, such as repeated motifs, ornaments, pauses, etc., are falsely identified as performance errors.

[0083] Such musical aspects allow humans to distinguish between genuine performance mistakes and mistakes in a musical context, even if the mistakes are superficially performed the same as the aforementioned mistakes. Thus, the learning model can be improved by better modeling the underlying composition. Therefore, to define the symptoms of noticeable errors, it is desirable to find a middle ground between the rule-based approach and the approach learned from empirical labels.

[0084] Thus, to define salient error signatures, it is desirable to find a middle ground between rule-based approaches and approaches learned from empirical labels. The result is a set of error descriptions, some of which occur at a particular moment in time, and others that occur over a longer window, whether that be a continuous window or a long period of intermittent labels.

[0085] However, because the conspicuity of errors is inspired by perceptual ideas, it is desirable that these errors be defined through an empirical process. Also, synthesis data can help improve performance. However, synthesis errors can sound unnatural in some cases. For example, when pitch insertion is induced, it is difficult to play with such confidence and tempo, despite the high number of out-of-context pitch insertions.

[0086] It has been observed that beginners make more complex errors and use recovery strategies than synthetic data that only generate simple errors such as repetitions, pauses, hesitations, and note insertions. Therefore, it is desirable to study the behavior of beginner pianists and create synthetic data with more natural errors.

[0087] 4, a temporal convolution network 40 is configured to receive data representative of a performance of an instrument to be evaluated. For example, in the illustrated embodiment, the temporal convolution network 40 receives MIDI data from the digital audio workstation 34.

[0088] The temporal convolutional network 40 is then executed by the processor 30 to identify errors in the musical performance using reference data stored in the computer-readable storage device 32 or a remote server 28 (e.g., a network server or a cloud server). In other words, in the illustrated example, the temporal convolutional network 40 detects intervals of such errors in the musical performance from the piano roll.

[0089] The results of the detection of errors by the temporal convolutional network 40 are then output to a classifier head 42 which determines whether the detected errors are considered to be noticeable or inconspicuous errors. As used herein, a "noticeable error" means an error that is easily recognizable by a listener of a given skill level without reference to or knowledge of musical notation. The "predetermined skill level" is based on the collected reference data and the skill level of the listener that is used to determine whether an error is a noticeable or inconspicuous error.

[0090] As described below, preferably, a user of the performance evaluation system 10 can selectively set a threshold that distinguishes between non-noticeable errors and noticeable errors. In this manner, a user of the performance evaluation system 10 can fine-tune the level of errors that the user is notified of.

[0091] Specifically herein, the digital audio workstation 34 outputs data representing at least one piano roll from a musical performance by the digital piano 12. Preferably, the digital audio workstation 34 outputs data representing a first piano roll 50 and a second piano roll 52 to the temporal convolution network 40. In other words, in the illustrated embodiment, for a given sequence of piano note events in the musical performance, two piano rolls are extracted.

[0092] The first piano roll 50 is for the onset of the note, and the second piano roll 52 is for the sustained portion of the key press. Specifically, a set of I MIDI note events (start time, end time, pitch, velocity) {(si,ei,pi,vi)} i I and the sampling rate R. Then, for s∈[si,ei], there is a 256-dimensional piano roll X∈ such that X(pi,round(R·si))=vi and X(128+pi,round(Rs))=vi. R256×T is calculated.

[0093] Then, a Python package such as Partitura is used for the calculations, with R set to 16Hz. Partitura can read sheet music (in MEI, MusicXML, Kem and MIDI formats), MIDI performances and score-performance alignments. The package includes several tools for music analysis, such as automatic pitch spelling, key signature identification and voice separation.

[0094] Additionally, sustain pedal information is ignored in the second piano roll 52 calculations. Since beginning pianists tend to keep the pedal pressed down, causing the calculated note lengths to become excessively long, it is preferable to ignore the sustain pedal information to avoid smearing the second piano roll 52 in the sustain section.

[0095] 5, there is shown a flow chart illustrating the evaluation process carried out by computing device 14 of performance evaluation system 10 using a temporal convolutional network 40 and a classifier head 42. In this example, the musical instrument is a digital piano 12, and computing device 14 includes a digital audio workstation 34 and performance evaluation program 36 software.

[0096] Essentially, the evaluation process in the flowchart of Figure 5 is as follows: Obtaining a musical performance of a user on an instrument; identifying errors in the musical performance using at least one processor 30 based on reference data stored in a computer readable storage device 32; classifying, using at least one processor 30, errors in the musical performance into either noticeable errors or non-noticeable errors; instructing a notification device (e.g., display 16 and / or speaker 22) to output an evaluation of the musical performance distinguishing between noticeable and non-noticeable errors; The present invention represents a computer-implemented method for evaluating a performance, comprising:

[0097] 5, in the evaluation process, a user launches a digital audio workstation 34 installed on computing device 14 and begins recording a musical performance by selecting "record." That is, in step S1 of the evaluation process, processor 30 captures at least a portion of the musical performance being received via an audio input such as communication cable 24 (digital connection) or microphone 26. Preferably, the evaluation process is performed by evaluating successive sections of the musical performance.

[0098] Next, in step S2 of the evaluation process, the processor 30 creates performance data indicative of the notes played by the musical instrument. Here, the processor 30 calculates the first piano roll 50 and the second piano roll 52 described above. Of course, the processor 30 creates the performance data in other formats as needed and / or desired. Here, the processor 30 utilizes the digital audio workstation 34 to calculate the first piano roll 50 and the second piano roll 52.

[0099] Next, in step S3 of the evaluation process, the processor 30 uses the temporal convolutional network 40 to identify errors in the musical performance. Essentially, the temporal convolutional network 40 compares the performance data (e.g., the first piano roll 50 and the second piano roll 52) with reference data stored in the computer-readable storage device 32 or a remote server 28 (e.g., a network server or a cloud server). In this manner, errors in the performance are identified.

[0100] Next, in step S4 of the evaluation process, processor 30 determines whether there are any noticeable errors in the musical performance, utilising classifier head 42. More specifically, the errors in the musical performance identified by temporal convolution network 40 are input to classifier head 42.

[0101] The classifier head 42 then uses the reference data to determine a probability of being a noticeable error for each error. The probability of being a noticeable error for each error is then compared to a threshold for classifying each error as either a noticeable error or a non-noticeable error. That is, if the probability of being a noticeable error is equal to or greater than the threshold, the error is determined to be a noticeable error. On the other hand, if the probability of being a noticeable error is less than or equal to the threshold, the error is determined to be a non-noticeable error.

[0102] The noticeable error threshold may be a single value for all types of errors or may be different values ​​for different types of errors, for example if the classifier head 42 determines that a particular error has a 50% probability of being a noticeable error and the threshold for that type of error is set at 39%, then the error will be determined to be a noticeable error.

[0103] Next, in step S5 of the evaluation process, the processor 30 outputs a notification of the noticeable errors in the musical performance, if any. The notification for each section of the musical performance may be output in real time (or near real time) or may be stored for later output. The notification from the processor 30 may be communicated to the performer using a notification device such as the display 16 or the speaker 22.

[0104] After performing step S5 of the evaluation process, processor 30 returns to step S1 to repeat the evaluation process for the next section of the musical performance. The performance is stopped when the performer or other person stops recording the performance.

[0105] When the first piano roll 50 and the second piano roll 52 for a given sequence of note events in a musical performance are input to the temporal convolutional network 40, the presence of errors in the musical performance is detected by comparing the first piano roll 50 and the second piano roll 52 with reference data that has been pre-trained and implemented in the temporal convolutional network 40 and the classifier head 42.

[0106] 7 shows an example of a piano roll 54 computed from a musical performance that is input to the temporal convolutional network 40 from the digital audio workstation 34. In the example piano roll 54 of FIG. 7, the musical performance includes several errors or mistakes made by the performer of the musical performance. Here, five errors are identified by the temporal convolutional network 40 for the example piano roll 54 as error "A", error "B", error "C", error "D", and error "E".

[0107] Referring to FIG. 8, error “A” is identified in the exemplary piano roll 54 as a harmonically conflicting note insertion error, which the temporal convolution network 40 and classifier head 42 identify as a noticeable error.

[0108] That is, the temporal convolution network 40 computes the score on the left side of Fig. 8 as the underlying score of the musical performance, and thus identifies errors in the performed musical performance (encircled by dashed lines) on the right side of Fig. 8. The classifier head 42 then determines the probability that the detected error is a noticeable error based on a pre-set threshold for note insertion errors.

[0109] Referring to FIG. 9, error “B” is identified in the exemplary piano roll 54 as a harmonically conflicting note modification error, which the temporal convolutional network 40 and classifier head 42 identify as a non-obvious error.

[0110] That is, the temporal convolution network 40 computes the score on the left side of Fig. 9 as the underlying score of the musical performance, and thus identifies the errors (encircled by dashed lines) in the performed musical performance on the right side of Fig. 9. The classifier head 42 then determines the probability that the detected error is a non-noticeable error based on a pre-set threshold for note correction errors.

[0111] Referring to FIG. 10, error “C” is identified in the exemplary piano roll 54 as a musical natural note modification error, which the temporal convolutional network 40 and classifier head 42 identify as a non-obtrusive error.

[0112] That is, the temporal convolution network 40 computes the score on the left side of Fig. 10 as the underlying score of the musical performance, and thus identifies the errors (encircled by dashed lines) in the performed musical performance on the right side of Fig. 10. The classifier head 42 then determines the probability that the detected error is a non-noticeable error based on a pre-set threshold for note correction errors.

[0113] 11, error "D" is identified in the exemplary piano roll 54 as an abrupt pause or repetition error, which the temporal convolution network 40 and classifier head 42 identify as a noticeable error.

[0114] That is, the temporal convolution network 40 computes the score on the left side of Fig. 11 as the underlying score of the musical performance, and thus identifies the errors (encircled by dashed lines) in the performed musical performance on the right side of Fig. 11. The classifier head 42 then determines the probability that the detected error is a noticeable error based on a pre-set threshold for sudden pause / repetition errors.

[0115] Referring to FIG. 12, error “E” is identified in the exemplary piano roll 54 as a rhythmically unnatural timing error, which the temporal convolution network 40 and classifier head 42 identify as a noticeable error.

[0116] That is, the temporal convolution network 40 computes the score on the left side of Fig. 12 as the underlying score of the musical performance, and therefore identifies the errors (encircled by dashed lines) in the performed musical performance on the right side of Fig. 12. The classifier head 42 then determines the probability that the detected error is a noticeable error based on a pre-set threshold for timing errors.

[0117] Next, a first notification method of the music performance evaluation system 10 will be described with reference to Figures 13 and 14. In the first notification method, the notification device includes a display 16 that presents an evaluation of the music performance.

[0118] The user now launches the digital audio workstation 34 installed on the computing device 14 and selects "record" to begin recording the musical performance. The processor 30 then begins capturing the musical performance as it is being received via an audio input, such as the communication cable 24 (digital connection) or the microphone 26.

[0119] The at least one processor 30 is configured to present a piano roll of the musical performance on the display 16 to distinguish between the presence of noticeable errors and the presence of non-noticeable errors. In particular, as the musical performance is being recorded, the at least one processor 30 using the digital audio workstation 34 creates a piano roll of the musical performance that is displayed on the screen of the display 16, as shown in FIG.

[0120] In the background, at least one processor 30 using a performance evaluation program 36 receives the piano MIDI roll and identifies noticeable errors in the musical performance. Preferably, the evaluation process is performed by evaluating successive sections of the musical performance. Alternatively, in this notification method, the evaluation process can be delayed until the user stops recording the performance or enters a command to generate an evaluation of the performance.

[0121] In any event, once the user stops recording the musical performance, performance evaluation program 36 generates an evaluation of the musical performance, either automatically or in response to a user-entered command to generate an evaluation of the musical performance.

[0122] An example of an evaluation of a musical performance is shown in Figure 14. As shown in Figure 14, a piano roll is displayed on the screen of the display 16 along with notable errors identified in the musical performance. In particular, the at least one processor 30 is configured in a first notification method to present only the notable errors in the piano roll presented by the display 16.

[0123] Here, as shown in Fig. 14, for example, a vertical line is superimposed on the piano roll displayed on the screen of the display 16 to indicate a noticeable error. In the first notification method, less noticeable errors are not identified. As will be apparent from this disclosure, noticeable errors can be identified in other ways on the piano roll, such as a callout explaining the error.

[0124] Additionally, the notification can be other types of visual indicators not including a piano roll, and if desired, the user can enter a command to display all errors (silent and noticeable) in the recorded performance.

[0125] Next, a second notification method of the performance evaluation system 10 will be described with reference to Figures 15 to 17. In the second notification method, the notification device includes a display 16 that presents the evaluation of the performance.

[0126] Here, similar to the first notification method, the user launches the digital audio workstation 34 installed on the computing device 14 and selects "record" to begin recording the musical performance. Thus, similar to the first notification method, the processor 30 begins capturing the musical performance, and the digital audio workstation 34 creates a piano roll of the musical performance that is displayed on the screen of the display 16 as illustrated in FIG.

[0127] However, in the second notification method, the piano roll is not used to indicate noticeable errors. The digital audio workstation 34 creates a score for the musical performance that is displayed on the screen of the display 16, as illustrated in FIG.

[0128] The at least one processor 30 is configured to display on the display 16 a score that identifies the presence of noticeable errors on the score as distinct from non-noticeable errors. In other words, the at least one processor 30 utilizes the performance assessment program 36 to identify the presence of noticeable errors in the musical performance.

[0129] The user can now select how to display the errors on the displayed music score: the user can choose to display all errors (non-obvious and obvious errors) in the recorded music performance, as seen in Figure 15, or they can choose to display only the obvious errors in the recorded music performance, as shown in Figure 16.

[0130] Additionally, the user may choose to display all errors in the musical performance, distinguishing between noticeable and non-noticeable errors, as in Figure 17. Thus, in the second notification method, at least one processor 30 is configured to present a score of the musical performance on display 16 distinguishing between noticeable errors and non-noticeable errors, and is configured to present only the noticeable errors on the score presented by display 16.

[0131] Next, a third notification method of the music performance evaluation system 10 will be described with reference to Figures 18 and 19. In the third notification method, the notification device includes a display 16 that presents an evaluation of the music performance.

[0132] However, in the third notification method, speaker 22 may be utilized as the notification device. Similar to the first notification method, the user launches digital audio workstation 34 installed on computing device 14 and selects "record" to begin recording the musical performance. Thus, similar to the first notification method, processor 30 begins capturing the musical performance and begins error evaluation of the musical performance.

[0133] Here, however, notification of noticeable errors is provided to the performer in real time or near real time - in other words, the at least one processor 30 is configured to present noticeable errors on the display 16 as the musical performance is being captured via an audio input (e.g., communication cable 24 or microphone 26). For example, a virtual listener avatar is displayed on the screen of the display 16 while the performance is taking place.

[0134] The virtual listener avatar is responsible for notifying the user of noticeable errors. At least one processor 30 is configured to use the virtual listener avatar to present on display 16 a virtual listener avatar that identifies the presence of noticeable errors when the musical performance is being captured by an audio input (e.g., communication cable 24 or microphone 26).

[0135] In particular, the at least one processor 30 is configured to change the virtual listener avatar from a first image of Figure 18 to a second image of Figure 19 to identify the presence of a noticeable error, where the first image illustrates the absence of a noticeable error at that point in the musical performance, and the second image illustrates a noticeable error at that point in the musical performance.

[0136] More specifically, when a noticeable error is identified during performance, the facial expression of the virtual listener avatar changes from a first expression (ie, a first notification) to a second expression (ie, a second notification).

[0137] For example, the first facial expression of the virtual listener avatar (i.e., the first notification) may be a happy, neutral, or indifferent facial expression indicating that there is no noticeable error, while the second facial expression of the virtual listener avatar (i.e., the second notification) may be a surprised facial expression indicating that there is a noticeable error.

[0138] Even in this case, if the user also desires to be notified of an inconspicuous error during performance, the performance evaluation program 36 can identify the presence of an inconspicuous error during performance with a facial expression different from the second facial expression of the virtual listener avatar. That is, the facial expression of the virtual listener avatar changes from the first facial expression to a third facial expression (i.e., a third notification) that is different from both the first facial expression and the second facial expression of the virtual listener avatar. For example, the third facial expression of the virtual listener avatar can be a confused facial expression indicating the presence of an inconspicuous error.

[0139] In understanding the scope of the present invention, the term "comprising" and its derivatives as used herein are intended to be open-ended terms that specify the presence of stated features, elements, components, groups, integers, and / or steps, but do not exclude the presence of other unstated features, elements, components, groups, integers, and / or steps. The above interpretation also applies to words of similar meaning, such as "comprising," "having," and their derivatives.

[0140] In addition, although the terms "first" and "second" may be used herein to describe various components, it will be understood that these components should not be limited by these terms. These terms are merely used to distinguish one component from another. Thus, for example, a first component described above could be referred to as a second component, and vice versa, without departing from the teachings of the present invention.

[0141] Although only selected embodiments have been chosen to illustrate the invention, it will be apparent to those skilled in the art from this disclosure that various changes and modifications can be made herein without departing from the scope of the invention as defined in the appended claims.

[0142] For example, unless specifically stated otherwise, the size, shape, location or orientation of various components may be changed as necessary and / or desired so long as such changes do not materially affect their intended function.

[0143] Unless otherwise specified, components shown directly connected or in contact with each other may have intermediate structures disposed therebetween so long as such changes do not materially affect their intended function. Unless otherwise specified, the function of one element may be performed by two elements, and vice versa.

[0144] Structures and features of one embodiment may be employed in other embodiments. Not all advantages need simultaneously be present in a particular embodiment. Any feature that is distinct from the prior art, either alone or in combination with other features, should be considered as a separate description of Applicant's further invention, including the structural and / or functional concepts embodied by such feature(s). Accordingly, the foregoing description of embodiments in accordance with the present invention is provided by way of example only and is not intended to limit the invention as defined by the appended claims and their equivalents. [Explanation of symbols]

[0145] 10... performance evaluation system, 12... digital piano, 14... computing device, 16... display, 18... keyboard, 20... mouse, 22... speaker, 24... communication cable, 26... microphone, 28... remote server, 30... processor, 32... storage device, 34... digital audio workstation, 36... performance evaluation program, 40... temporal convolutional network, 42... classifier head, 44... autoencoder, 46... encoder, 48... decoder, 50... first piano roll, 52... second piano roll, 54... piano roll,

Claims

1. an audio input configured to input a musical performance; a notification device configured to output a rating of the music performance; at least one processor connected to the notification device; Equipped with The at least one processor executes a performance evaluation program to Identifying errors in musical performance based on reference data; classifying the errors in the musical performance into either noticeable errors or inconspicuous errors; instructing the notification device to output an evaluation of the music performance by distinguishing between the noticeable errors and the non-noticeable errors. Performance evaluation system.

2. The at least one processor is configured to instruct the notification device to output an evaluation of the musical performance upon identifying only the presence of the noticeable error. The performance evaluation system according to claim 1.

3. a digital audio workstation for recording said musical performance and for providing MIDI note events of said musical performance as input to said at least one processor.

2. The performance evaluation system of claim 1, further comprising:

4. The MIDI note events of the musical performance include, for each note event of the musical performance, a start time, an end time, a pitch, and a velocity. The performance evaluation system according to claim 3.

5. The digital audio workstation generates at least one piano roll of the musical performance as an input to the at least one processor. The performance evaluation system according to claim 4.

6. The at least one piano roll comprises: a first piano roll providing data representative of the onset of notes in the musical performance; and a second piano roll for providing data representative of sustained portions of key depressions in said musical performance. The performance evaluation system according to claim 5.

7. The at least one processor utilizes a learning model in the performance evaluation program to classify errors in the musical performance. The performance evaluation system according to claim 1.

8. The learning model includes a temporal convolutional network. The performance evaluation system according to claim 7.

9. The learning model is The method further includes a classifier head provided after the temporal convolution network. The performance evaluation system according to claim 8.

10. The notification device includes a display that displays a rating of the musical performance. The performance evaluation system according to claim 1.

11. The at least one processor: configured to display on the display a piano roll of the musical performance that distinguishes between the noticeable errors and the non-noticeable errors. The performance evaluation system according to claim 10.

12. The at least one processor: and configured to present only the noticeable errors on the piano roll displayed by the display. The performance evaluation system according to claim 11.

13. The at least one processor: and displaying the score of the musical performance on the display, distinguishing between the noticeable errors and the non-noticeable errors on the score of the musical performance. The performance evaluation system according to claim 10.

14. The at least one processor: and configured to display only the noticeable errors on the music score displayed on the display. The performance evaluation system according to claim 13.

15. The at least one processor: configured to display the noticeable error on the display when the musical performance is captured by the audio input. The performance evaluation system according to claim 10.

16. The at least one processor: and configured to display, on the display, a virtual listener avatar that identifies the noticeable errors when the musical performance is captured with the audio input. The performance evaluation system according to claim 15.

17. The at least one processor: configured to change the virtual listener avatar from a first image to a second image to identify the presence of the noticeable error; the first image showing the absence of the noticeable error at that point in the musical performance; The second image shows the noticeable error at that point in the musical performance. The performance evaluation system according to claim 16.

18. the first image of the virtual listener avatar is a first facial expression; The second image of the virtual listener avatar is a second facial expression. The performance evaluation system according to claim 17.

19. Obtaining a musical performance by a user; utilizing at least one processor to identify errors in the musical performance based on reference data; classifying, utilizing the at least one processor, the errors in the musical performance as either noticeable errors or non-noticeable errors; instructing a notification device to output an evaluation of the music performance by distinguishing between the noticeable errors and the non-noticeable errors; A computer-implemented performance evaluation method comprising:

20. A non-transitory computer-readable recording medium storing a performance evaluation program, the performance evaluation program being executed by a computing device to: Obtaining a musical performance by a user; utilizing at least one processor of the computing device to identify errors in the musical performance based on reference data; utilizing the at least one processor of the computing device to classify the errors in the musical performance as either noticeable errors or non-noticeable errors; instructing a notification device to output an evaluation of the music performance by distinguishing between the noticeable errors and the non-noticeable errors; A recording medium for causing the computing device to execute the above.

Images