Performance difference detection method, program and system, and performance error part notification method

Abstract

To provide a performance difference detection method which can easily, quickly, and concretely identify a difference between basic performance data which a teacher performs and determination performance data which a student performs with a bird's-eye view which the student can intuitively easily understand.SOLUTION: A fluctuation imparting part 13 imparts fluctuation at each note-on time and note-off time of performance data of basic performance, generates a plurality of pieces of sub-basic performance data and forms a sub-basic performance data cluster MDTC. A voice data conversion part 14 and an image data conversion part 15 generate a sub-basic performance image data cluster IDTC. An error detection part 21 performs learning with each sub-basic performance image data of the sub-basic performance image data cluster as teacher data. Each of the conversion parts 14 and 15 converts determination performance data MDS into determination performance image data IDS and an error part on determination performance image data IDS is marked M by error detection in the learned error detection part 22.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a performance difference detection method, a program, a system, and a performance error notification method, and more particularly to a performance difference detection method, a program, a system, and a performance error notification method that use computer processing to find the difference between a basic performance played by a teacher or the like and a judged performance played by a student or the like. [Background technology]

[0002] For example, when learning to play the piano, it is desirable to find the difference between the basic performance data played by the teacher and the judged performance data played by the student. Prior art for finding such a difference is described in the following Patent Documents 1 and 2. The technology of Patent Document 1 performs "individual comparison of specific sounds" by obtaining a difference value between certain values ​​of two sounds, obtaining information indicating the magnitude of the obtained difference value, and outputting the obtained information to an alarm unit. In the technology of Patent Document 2, based on "multiple performance operations," at least one of the speed and tonality of the performance operations, in which multiple performance operations within a set time are treated as one performance operation, is evaluated, and a comparison is made in which an output means is instructed to output a reaction sound corresponding to the evaluation depending on the timing of the evaluation during the performance. In all of the above-mentioned conventional techniques, it is necessary to identify and compare individual sounds, and it is necessary to have multiple performances that serve as the basis for comparison, and there is no simple, quick, and comprehensive comparison of the entire performance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-089387 [Patent Document 2] Patent Publication No. 2023-034873 Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the current situation, the present invention aims to provide a performance difference detection method, program, and system, as well as a performance error notification method, which are capable of easily, quickly, and specifically pointing out the differences between basic performance data played by a teacher or the like and judgment performance data played by a student or the like, with an overview that allows the student to easily understand the differences intuitively. [Means for solving the problem]

[0005] In order to achieve the above object, a performance difference detection method according to the present invention is characterized in that, in a method for determining a difference between a basic performance and a judgment performance by computer processing, performance data of the basic performance has at least a key number that specifies a pitch, a note-on time that is the time when the sound of the key number starts, and a note-off time that is the time when the sound of the key number ends, and the method has a fluctuation imparting section, an audio data conversion section, an image data conversion section, and an error detection section, the fluctuation imparting section imparts fluctuation to the note-on time and the note-off time according to a predetermined rule, the audio data conversion section converts the performance data into audio data, and the image data conversion section converts the audio data into performance image data of a power spectrum with time and frequency as axes. The fluctuation adding unit adds fluctuation to each note-on time and note-off time of the basic performance data, creates multiple sub-basic performance data to form a sub-basic performance data cluster, converts each of the sub-basic performance data into each sub-basic audio data by the audio data conversion unit, and further converts them into each sub-basic performance image data by the image data conversion unit to create a sub-basic performance image data cluster, and the error detection unit learns using each of the sub-basic performance image data of the sub-basic performance image data cluster as teacher data, converts the judged performance into judged audio data, and further converts them into judged performance image data by the image data conversion unit, and marks error locations on the judged performance image data by the error detection in the learned error detection unit.

[0006] The conversion units convert the performance data into audio data and image data in sequence. The fluctuation imparting unit imparts fluctuation to the note-on time and the note-off time according to a predetermined rule, and a sub-basic performance data cluster, a sub-basic audio data cluster, and a sub-basic image data cluster are generated in sequence. The error detection unit performs learning using the data of the sub-basic image data cluster, and marks the error location on the determined performance image data by the error detection unit that has already learned. If the fluctuation width is set large in the fluctuation adding section, the difference width of the correct answer in the error detection becomes large, so that, for example, even if the student is a beginner, only major errors will be marked. On the other hand, if the fluctuation width is set small in the fluctuation adding section, the difference width of the correct answer in the error detection becomes narrow, so that, for example, even if the student is an advanced student, small errors will also be marked. Therefore, it is possible to point out and mark the major errors from a bird's-eye view throughout the entire song. Furthermore, since the error locations on the judged performance image data are marked using a power spectrum having a time axis and a frequency axis, the student being judged can visually know which parts of the performance are incorrect.

[0007] In the above configuration, the performance data may further include a performance intensity, and the fluctuation imparting unit may impart a fluctuation to the performance intensity as well. With the above configuration, it is also possible to perform a difference determination for an instrument such as a piano that uses keystroke strength as an expression element.

[0008] In this configuration, the performance data is MIDI data, the image data conversion unit performs mel spectrogram conversion, and the error detection unit detects errors by detecting anomalies in images using a convolutional neural network. The mel spectrogram conversion is a feature value corresponding to the human hearing characteristics for frequencies, and the overview is more in line with human hearing.

[0009] Also, for example, the basic performance data is data that has been performed and recorded by a teacher, and the judgment performance data is data that has been performed and recorded by a student.

[0010] In order to achieve the above-mentioned object, a feature of the performance error notification method of the present invention is that, in addition to the above-mentioned method, the time position marked at the error location on the judged performance image data is displayed, and further the error location is notified by playing the judged audio data.

[0011] In order to achieve the above-mentioned object, another feature of the performance error notifying method of the present invention is, in addition to the above-mentioned method, to display at least the time position marked at the error point on the judged performance image data on the musical score associated with the judged performance image data, thereby notifying the error point.

[0012] The present invention also provides a program for causing a computer to execute any of the above-described performance difference detection methods.

[0013] On the other hand, in order to achieve the above object, a performance difference detection system according to the present invention is characterized in that, in a configuration in which the difference between a basic performance and a judgment performance is found by computer processing, data of the basic performance has at least a key number that specifies the pitch, a note-on time that is the time when the sound of the key number starts, and a note-off time that is the time when the sound of the key number ends, and the system has a fluctuation imparting section, an audio data conversion section, an image data conversion section, and an error detection section, the fluctuation imparting section imparts fluctuation to the note-on time and the note-off time in accordance with a predetermined rule, the audio data conversion section converts the performance data into audio data, and the image data conversion section converts the audio data into a power spectrum having time and frequency axes. The fluctuation adding unit adds fluctuation to each note-on time and note-off time of the basic performance data, creates a plurality of sub-basic performance data to form a sub-basic performance data cluster, each of the sub-basic performance data is converted into each sub-basic performance image data by the audio data conversion unit and the image data conversion unit to create a sub-basic performance image data cluster, the error detection unit learns using each of the sub-basic performance image data of the sub-basic performance image data cluster as teacher data, converts the judged performance into judged audio data, and further converts it into judged performance image data by the image data conversion unit, and marks the erroneous part on the judged performance image data by error detection in the learned error detection unit. Effect of the Invention

[0014] According to the features of the performance difference detection method, program and system, and performance error notification method of the present invention, by changing the fluctuation width according to the level of the students, etc., it is possible to detect and display performance differences that can be easily, quickly and specifically pointed out, with an overview that allows students to easily and intuitively understand the differences between the basic performance data played by the teacher, etc. and the judgment performance data played by the students, etc.

[0015] Other objects, configurations and effects of the present invention will become apparent from the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram showing the overall configuration of a system for carrying out a method of the present invention. [Diagram 2] FIG. 2 is a diagram showing a flow of data processing using the system of FIG. 1. [Diagram 3] These are diagrams showing performance data to explain how to add fluctuations to MIDI data. (a) is the performance data (MIDI data) of a model performance of K545 (Sonata in C major). (b) is an enlarged view of a portion of (a). (c) is a diagram of (b) with fluctuations added to each note. (d) is the performance data with fluctuations added. [Figure 4] FIG. 13 is a diagram illustrating an example of a method for displaying anomaly detection. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present invention will now be described in more detail with reference to the accompanying drawings, in which: Fig. 1 is a diagram showing the overall configuration of a system for carrying out the method of the present invention, and Fig. 2 is a diagram showing the flow of data processing using the system of Fig. 1.

[0018] The performance difference detection system 1 comprises an image data generation block 10, a difference detection block 20 (error detection section), and a difference display block 30. The image data generation block 10 also comprises a MIDI module 12 and an interface 11 such as a keyboard connected thereto, a fluctuation generator 13 that is used as needed, and an audio data conversion section 14 and an image data conversion section 15 that are used in sequence.

[0019] An overview of the entire process will be explained first. Performance data MD input to a MIDI module 12 from an interface 11 such as a keyboard is converted into audio data WD with an extension such as wav by an audio data conversion unit 14, and the audio data WD is converted into image data ID by an image data conversion unit 15. A fluctuation generator 13 generates a large number of sub-basic performance data MDT1, 2, 3... by adding fluctuations to basic performance data MDTM (teacher) as described below, and creates a sub-basic performance data cluster MDTC. These are converted in sequence to create a sub-basic image data cluster IDTC, and the difference detection block 20 (error detection unit) is trained using this cluster as teacher data to perform error judgment on judgment image data IDS (student).

[0020] The behavior of each device will be described below. The MIDI module 12 is a sound source module created in compliance with the MIDI (Musical Instrument Digital Interface) standard, which is a common standard for transferring and sharing performance data of electronic musical instruments between devices. The interface 11 is a piano keyboard in this embodiment, and can store and play music having multiple notes. The MIDI module 12 has at least a key number that specifies the pitch, a note-on time that is the time when the sound of the key number starts, and a note-off time that is the time when the sound of the key number ends. Furthermore, the keyboard has an acceleration sensor and has the keystroke strength (velocity) of each key, which allows the keystroke strength to be stored.

[0021] As shown in FIG. 3, the fluctuation generator (fluctuation imparting unit) 13 imparts fluctuation to each note N1-3 of the performance data MD stored in the MIDI module 12 according to a certain probability distribution. In the example of FIG. 3, for example, a fluctuation of a keying time (keying timing) of up to ±20 ms is imparted to the intermediate time between the note-on time t1a and the note-off time t1b of the first note N1, resulting in a sub-first note N1' (sub-note-on time t1a' and sub-note-off time t1b'). Fluctuation is also imparted to the other notes N2, N3, etc., resulting in a sub-second note N2' and a sub-third note N3'. In this way, a plurality of sub-basic performance data MDT1, 2, 3... are generated from the basic performance data MDTM (teacher), and a sub-basic performance data cluster MDTC is formed. The fluctuation is imparted to create learning data.

[0022] In the above example, as shown in Figures 2 and 3, for example, fluctuation is added to the intermediate time between the note-on time t1a and the note-off time t1b of the first sound N1, and the first data set DS1 is generated. However, this fluctuation may be added to the keying strength recorded for each sound, as shown in Figure 2, in which case the second data set DS2 is generated. Also, fluctuation may be added individually to the note-on time t1a and the note-off time t1b of the first sound N1. The probability distribution of the fluctuation may follow a normal distribution or any other continuous probability distribution.

[0023] The audio data conversion unit 14 is a player of the performance data MD, and converts MIDI data into audio data WD with the extension wav, etc. Chords can also be played by playing each key number sound at the speed (volume) recorded at the note-on time with the piano tone of the sound source at a pitch based on the key number and cutting off at the note-off time in parallel for multiple key numbers.

[0024] The image data conversion unit 15 converts the sound data WD into performance image data ID of a power spectrum with the time and frequency axes as axes. Here, mel spectrogram conversion is used. The composite signal consisting of sound data is passed through a window function to calculate the frequency spectrum, and the result is expressed as time on the horizontal axis, frequency on the vertical axis, and color as the strength of the signal components, and features corresponding to the human hearing characteristics for frequencies are used.

[0025] The MIDI module 12, fluctuation generator 13, audio data conversion unit 14 and image data conversion unit 15 of the image data generation block 10 may be configured on the same or separate personal computers, or on the same or separate embedded computers, or may be realized by an API (Application Programming Interface) via the cloud.

[0026] In this way, using the functions of the image data generation block 10, a sub-basic performance data cluster MDTC consisting of sub-basic performance data MDT1, 2, 3 ... is created from the basic performance data MDTM (teacher), a sub-basic audio data cluster WDTC consisting of sub-basic audio data WDT1, 2, 3 ... is created, and finally a sub-basic image data cluster IDTC consisting of sub-basic image data IDT1, 2, 3 ... is created, and these are used as teacher data for machine learning.

[0027] On the other hand, for the judged performance data MDS of the student's performance, the fluctuation generator 13 is not used, and judged (student) voice data WDS and judged (student) image data IDS are created and used as the judgement target. The student's performance is converted into judged voice data WDS via the MIDI module 12 and the voice data converter 14, or the sound generated by using a piano or the MIDI module 12 in combination with a sound source may be recorded by a microphone or the like and converted into the judged voice data WDS.

[0028] The difference detection block 20 (error detection unit) may be realized by a personal computer or an API (Application Programming Interface) via the cloud. The error detection in this block uses anomaly detection from images using a convolutional neural network. In the following embodiment, PatchCore (ResNet50) is used.

[0029] First, the sub-basic image data IDT1, 2, 3, ... of the sub-basic image data cluster IDTC are uploaded in sequence to the pre-learning neural network 21, and machine learning is performed. This forms a neural network optimized for image error detection. Both the data set DS1 to which fluctuations in keystroke time are added and the data set DS2 to which fluctuations in keystroke strength are added are used for machine learning.

[0030] The judged (student) image data IDS is uploaded to the trained neural network 22 and judged. If there is no difference according to the definition of error detection, no error is pointed out, and for the part where there is a difference, a marking is displayed on the judged (student) image data IDS using the image display unit 31 of the difference display block 30, as shown in Figures 2 and 4.

[0031] Regarding marking, as shown in Figs. 1 and 4, the judged (student) image data IDS has a vertical axis of frequency (higher frequency at the bottom) and a horizontal axis of time, with the sound intensity at each frequency displayed in color (although the figure shows black and white, color display is the basis). Therefore, the substantial difference (difference within a certain probability distribution) due to image recognition with the basic (teacher) performance data MDTM can be marked with markers M and M', and the student who is to be judged can recognize which notes on the score are different from the performance of the teacher, who is the basic performer, based on the coordinates of the frequency domain and time. Note that the markers may be displayed as symbol M' for only the different parts of the image, or as symbol M for the range including symbol M'.

[0032] The audio file linking unit 32 of the difference display block 30 links the judged image data IDS with the judged audio data WDS, outputs audio from the audio output unit 33, and displays the currently played part with a performance marker 32b on the performance bar 32a. The performance data linking unit 34 links the judged audio data WDS with the judged performance data MD, and generates and displays a score MS from the judged performance data MD in the score display unit 35. At this time, the closest note that is different from the performance of the teacher, who is the basic performer, is marked with a marker M based on the coordinates of the frequency domain and time, so that the student, who is the subject of judgment, can recognize which score MS the note is on. The score MS may be generated from the teacher's basic performance data MDTM.

[0033] The present invention can be modified as needed, and the error detection algorithm and fluctuation application method can be other than those described above. [Industrial Applicability]

[0034] The present invention can be used as a performance difference detection method, program, and system. It may be applicable to keyboard instruments such as pianos and synthesizers, as well as string instruments such as guitar synthesizers that are provided with a pitch controller that keeps each sound constant. The keystroke time generally refers to the performance time of a keyboard or strings, and the keystroke strength generally refers to the performance strength of a keyboard or strings. [Explanation of symbols]

[0035] 1: performance difference detection system, 10: image data generation block, 11: interface, 12: MIDI module, 13: fluctuation generator (fluctuation adding section), 14: audio data conversion section, 15: image data conversion section, 20: difference detection block (error detection section), 21: pre-training neural network, 22: trained neural network, 30: difference display block, 31: image display section, 32: audio file linking section, 32a: performance bar, 32b: performance marker, 33: audio output section, 34: performance data linking section, 35: music score display section, M, M': Ma player, MS: music score, MD: performance data, WD: audio data, ID: image data, MDTM: basic (teacher) performance data, MDT1, 2, 3...: sub-basic performance data, MDTC: sub-basic performance data cluster, WDT1, 2, 3...: sub-basic audio data, WDTC: sub-basic audio data cluster, IDT1, 2, 3...: sub-basic image data, IDTC: sub-basic image data cluster, DS1: first data set, DS2: second data set, MDS: judged (student) performance data, WDS: judged (student) audio data, IDS: judged (student) image data

Claims

1. A performance difference detection method for determining a difference between a basic performance and a judged performance by computer processing, comprising the steps of: the performance data of the basic performance includes at least a key number that specifies a pitch, a note-on time that is the time when the sound of the key number starts, and a note-off time that is the time when the sound of the key number ends, The apparatus includes a fluctuation adding unit, a voice data converting unit, an image data converting unit, and an error detecting unit, the fluctuation imparting unit imparts fluctuation to the note-on times and the note-off times according to a predetermined rule, the audio data conversion unit converts the performance data into audio data; the image data conversion unit converts the audio data into performance image data having a power spectrum with time and frequency axes as axes; the fluctuation imparting unit imparts fluctuation to each note-on time and note-off time of the basic performance data, creates a plurality of sub-basic performance data, and forms a sub-basic performance data cluster; Each of the sub-basic performance data is converted by the audio data conversion unit into each sub-basic audio data, and further converted by the image data conversion unit into each sub-basic performance image data to create a sub-basic performance image data cluster; the error detection unit performs learning using each of the auxiliary basic performance image data of the auxiliary basic performance image data clusters as teacher data; A performance difference detection method in which the judged performance is converted into judged audio data, and further converted into judged performance image data by the image data conversion unit, and error locations on the judged performance image data are marked by the error detection unit that has been trained.

2. 2. The performance difference detection method according to claim 1, wherein the performance data further includes a performance intensity, and the fluctuation imparting section imparts a fluctuation to the performance intensity as well.

3. 2. The performance difference detection method according to claim 1, wherein the performance data is MIDI data, the image data conversion unit performs mel spectrogram conversion, and the error detection unit detects errors by detecting abnormalities from images using a convolutional neural network.

4. 2. The method for detecting differences in performance according to claim 1, wherein the basic performance data is data that has been performed and recorded by a teacher, and the judgment performance data is data that has been performed and recorded by a student.

5. A performance error notifying method for notifying the error part by displaying a time position marked at the error part on the judged performance image data and further playing back judged audio data, in addition to the performance difference detection method according to any one of claims 1 to 4.

6. In addition to the performance difference detection method described in any one of claims 1 to 4, a performance error notifying method is provided, which displays at least the time position marked on the error part on the judged performance image data on a musical score associated with the judged performance image data, and notifies the error part.

7. A program for causing a computer to execute the performance difference detection method according to any one of claims 1 to 4.

8. A performance difference detection system that uses computer processing to find the difference between a basic performance and a judged performance. the performance data of the basic performance includes at least a key number that specifies a pitch, a note-on time that is the time when the sound of the key number starts, and a note-off time that is the time when the sound of the key number ends, The apparatus includes a fluctuation adding unit, a voice data converting unit, an image data converting unit, and an error detecting unit, the fluctuation imparting unit imparts fluctuation to the note-on times and the note-off times according to a predetermined rule, the audio data conversion unit converts the performance data into audio data; the image data conversion unit converts the audio data into performance image data having a power spectrum with time and frequency axes as axes; the fluctuation imparting unit imparts fluctuation to each note-on time and note-off time of the basic performance data, creates a plurality of sub-basic performance data, and forms a sub-basic performance data cluster; Each of the auxiliary basic performance data is converted into each of the auxiliary basic performance image data by the audio data conversion unit and the image data conversion unit to generate an auxiliary basic performance image data cluster; the error detection unit performs learning using each of the auxiliary basic performance image data of the auxiliary basic performance image data clusters as teacher data; A performance difference detection system that converts the judged performance into judged audio data, and further converts it into judged performance image data by the image data conversion unit, and marks error parts on the judged performance image data by error detection in the learned error detection unit.

Images