Musical performance auxiliary device, musical performance auxiliary method and musical performance auxiliary program

Abstract

To provide a technique capable of highly accurately detecting both a constituent sound miss and a timing miss of a musical instrument performance.SOLUTION: A musical performance miss detection part 14 extracts time-series first feature quantity data representing a feature quantity of a musical instrument performance and respective time-series second feature quantity data representing respective feature quantities of a model musical performance and a miss musical performance, compares respective distances between the first feature quantity data and the respective second feature quantity data through AI, and detects the model musical performance or miss musical performance associated with second feature quantity data closest in distance to the first feature quantity data so as to detect whether there is a constituent sound miss and its position in the musical instrument performance. The musical performance miss detection part also extracts time-series first sound pressure data of a musical instrument performance and time series second sound pressure data of the model musical performance, detects respective positions of sound pressure in corresponding relation between the first sound pressure data and second sound pressure data and time differences between the positions through AI, and determines whether the detected position and time difference in the musical instrument performance correspond to a position and a time difference in the miss musical performance so as to detect whether there is a timing miss in the musical instrument performance and its position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an assistive playing device, an assistive playing method, and an assistive playing program. [Background technology]

[0002] When students of piano lessons practice at home, it is difficult for them to recognize mistakes in their own playing. Therefore, there is a demand for technology that can detect mistakes in playing and advise students on areas to correct, thereby assisting students in their practice. One such technology is to record performances with a microphone and score them (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Soprano Recorder Lessons Plus”, Yamaha Corporation, [online], [searched on July 10, 2023],<URL: https: / / ses.yamaha.com / products / digital_soprano_plus / > Summary of the Invention [Problem to be solved by the invention]

[0004] However, with instruments such as the piano, which can produce multiple notes simultaneously, it has been difficult to recognize the constituent notes from microphone recordings and detect performance errors.

[0005] The present disclosure has been made in consideration of the above, and aims to provide a technique capable of detecting both mistakes in constituent notes and mistakes in timing when playing a musical instrument with a high degree of accuracy. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided an assistive music playing device that assists in playing a musical instrument, the assistive music playing device comprising: a registration unit that registers each piece of score data of a model performance and an erroneous performance in a storage unit; a reception unit that receives recording data of the musical instrument performance; and a detection unit that uses the score data of the model performance and the erroneous performance and the recording data of the musical instrument performance to detect mistakes in the musical instrument performance using AI, wherein the detection unit extracts first feature amount data in a time series that represents a feature amount of the musical instrument performance and each piece of second feature amount data in a time series that represents each feature amount of the model performance and the erroneous performance, compares the magnitude of each distance between the first feature amount data and each piece of second feature amount data using AI that measures the similarity between the time series data, and determines whether the first feature amount data is closest to the first feature amount data. By detecting model performances or erroneous performances related to the second feature data, the presence or absence of constituent note mistakes in the musical instrument performance and, if there are constituent note mistakes, the positions of the constituent note mistakes are detected; first time-series sound pressure data of the musical instrument performance and second time-series sound pressure data of the model performance are extracted; each position of sound pressure that corresponds between the first sound pressure data and the second sound pressure data and the time difference between the positions are detected using an AI that measures the similarity between time-series data taking into account the shape of the waveforms; and by determining whether the detected positions and time differences in the musical instrument performance correspond to the positions and time differences in the erroneous performance, the presence or absence of timing mistakes in the musical instrument performance and, if there are timing mistakes, the positions of the timing mistakes are detected.

[0007] According to one aspect of the present disclosure, there is provided a performance assistance method for assisting a musical instrument player, the performance assistance device performing the following steps: a first step of registering score data for model performances and erroneous performances in a storage unit; a second step of receiving recording data of the musical instrument performance; and a third step of detecting mistakes in the musical instrument performance using the score data for the model performance and the erroneous performances and the recording data of the musical instrument performance by an AI. In the third step, first feature data in a time series representing a feature of the musical instrument performance and each of second feature data in a time series representing each of the feature amounts of the model performance and the erroneous performances are extracted, and a distance between the first feature data and each of the second feature data is compared by an AI that measures a similarity between time-series data. By detecting the model performance or erroneous performance related to the second feature data that is closest to the data, the presence or absence of a constituent note mistake in the musical instrument performance and, if a constituent note mistake exists, the position of the constituent note mistake is detected; first time-series sound pressure data of the musical instrument performance and second time-series sound pressure data of the model performance are extracted; each position of sound pressure that corresponds between the first sound pressure data and the second sound pressure data and the time difference between the positions are detected using an AI that measures the similarity between time-series data taking into account the shape of the waveform; and by determining whether the detected positions and time differences in the musical instrument performance correspond to the positions and time differences in the erroneous performance, the presence or absence of a timing mistake in the musical instrument performance and, if a timing mistake exists, the position of the timing mistake is detected.

[0008] An assistance playing program according to one aspect of the present disclosure causes a computer to function as the assistance playing device. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a technique capable of detecting both mistakes in the constituent notes and mistakes in timing when playing a musical instrument with a high degree of accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a functional block configuration of a performance sound scoring device. [Diagram 2] FIG. 2 is a diagram showing the overall processing flow. [Diagram 3] FIG. 3 is a diagram showing an example of musical score data of a model performance. [Figure 4] FIG. 4 is a diagram showing an example of sheet music data of a performance error (missed constituent notes). [Diagram 5] FIG. 5 is a diagram showing an example of musical score data of a performance error (timing error). [Figure 6] FIG. 6 is a diagram showing a processing flow for detecting a performance error. [Figure 7] FIG. 7 is a diagram showing a method for generating time-series feature amount data of a model performance. [Figure 8] FIG. 8 is a diagram showing an example of time-series feature amount data of incorrect performance. [Figure 9] FIG. 9 is a diagram showing a method for generating time-series feature amount data of a musical instrument performance. [Figure 10] FIG. 10 is a diagram showing an image of a path detected between the feature amount data of a model performance and the feature amount data of a correct musical score performance. [Figure 11] FIG. 11 is a diagram showing the determination results of performance errors. [Figure 12] FIG. 12 is a diagram showing a processing flow for detecting a performance error. [Figure 13] FIG. 13 is a diagram showing a processing flow for detecting a timing error. [Figure 14] FIG. 14 is a diagram showing an example of sound pressure data of a model performance and a musical instrument performance. [Figure 15] FIG. 15 is a diagram showing an example of setting conditions for a timing error. [Figure 16] FIG. 16 is a diagram showing the determination result of a performance error. [Figure 17] FIG. 17 is a diagram showing a hardware configuration of a performance sound scoring device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0012] [overview] In piano classes, lessons are held once a week for about 30 minutes, and practice pieces from textbooks such as Burgmuller's are taught. Students are assigned homework by the teacher and practice at home every day. However, since the teacher is not present, students cannot judge the quality of their performance. Beginners often do not notice mistakes in their playing, which reduces the effect of practicing at home.

[0013] Therefore, this disclosure discloses a technology to assist playing during home practice, etc. Specifically, the technology uses AI to detect mistakes made during home practice, etc. In particular, it detects both mistakes in the constituent notes and mistakes in timing of playing with a high degree of accuracy. It also detects mistakes made when playing acoustic instruments, such as the piano, which are not electronic instruments.

[0014] Note that the piano is an example of a musical instrument. A piano class is an example of an application case of the present disclosure. Practicing at home is an example of a performance. The present disclosure is not limited to pianos, piano classes, and practicing at home, but can be applied to various cases. For example, the present disclosure can be applied to a flute instead of a piano. The present disclosure can be applied to various instruments such as keyboard instruments, percussion instruments, string instruments, and wind instruments. The present disclosure can be applied to a karaoke hall instead of a home. The present disclosure can be applied to a performance instead of a practice.

[0015] [Configuration of the performance sound scoring device] FIG. 1 is a diagram showing a functional block configuration of a performance sound scoring device 1 according to the present embodiment.

[0016] The performance scoring device 1 includes a musical score data registration unit 11, a musical score data storage unit 12, a recording data reception unit 13, a performance error detection unit 14, and a performance scoring unit 15.

[0017] The score data registration unit 11, the score data storage unit 12, the recording data reception unit 13, and the performance error detection unit 14 constitute an assistive playing device that assists in playing a musical instrument.

[0018] The score data registration unit (registration unit) 11 has a function of registering the score data of the model performance and the score data of the incorrect performance in the score data storage unit 12.

[0019] The score data storage unit (storage unit) 12 has a function of storing the score data of the model performance and the score data of the incorrect performance registered by the score data registration unit 11.

[0020] The recording data receiving unit (receiving unit) 13 has a function of receiving recording data of a musical instrument performance performed by a user.

[0021] The performance error detection unit 14 (detection unit) has a function of detecting errors in the user's musical instrument performance using AI (Artificial Intelligence) by using the score data of the model performance, the score data of the incorrect performance, and the recording data of the musical instrument performance.

[0022] For example, the performance error detection unit 14 detects the presence or absence of a component note error in an instrument performance and, if there is a component note error, the position of the component note error, using an AI that measures the similarity between time-series data taking into account the shape of the waveform.The performance error detection unit 14 detects the presence or absence of a timing error in an instrument performance and, if there is a timing error, the position of the timing error, using an AI that measures the similarity between time-series data taking into account the shape of the waveform.

[0023] The performance sound scoring unit 15 has a function of scoring the performance sound of a musical instrument using AI based on the detection results of mistakes in the musical instrument performance (mistakes in constituent notes, timing mistakes).

[0024] [Operation of the performance scoring device] The overall operation of the performance scoring device 1 will be outlined first, and then each individual operation will be explained.

[0025] As an example of "AI that measures the similarity between time series data", we use the DTW (Dynamic Time Warping) method of DP (Dynamic Programming) matching. As an example of "AI that measures the similarity between time series data taking into account the waveform shape", we use the DDTW (Derivative DTW) method.

[0026] FIG. 2 is a diagram showing the overall process flow performed by the performance sound scoring device 1.

[0027] The musical score data registration unit 11 accepts registration of model performance musical score data and erroneous performance musical score data input by a user, and stores the accepted model performance musical score data and erroneous performance musical score data in the musical score data storage unit 12 (step S101).

[0028] Next, the recording data receiving unit 13 receives recording data of a musical instrument performance performed by a user for the same piece of music as the model performance (step S102).

[0029] Next, the performance error detection unit 14 reads out the score data of the model performance and the score data of the incorrect performance from the score data storage unit 12, extracts each piece of time-series feature data for each performance and instrument performance, and compares each other's feature data using the DTW method of DP matching to detect whether or not there is a component note error due to a mistake in pressing a piano key, etc., and if there is a component note error, the position of the component note error (step S103).

[0030] Next, the performance error detection unit 14 reads out the score data of the model performance from the score data storage unit 12, extracts each of the time-series sound pressure data for the model performance and the instrument performance, compares each other's sound pressure data using the DDTW method to detect the difference in the timing of pressing the piano keys, etc., and compares this timing difference with the timing difference of the incorrect performance to detect whether there is a timing error in pressing the piano keys, etc., and if there is a timing error, the position of the timing error (step S104).

[0031] Finally, the performance sound scoring unit 15 scores the performance sound of the musical instrument performance using AI based on the component note mistakes of the musical instrument performance detected in step S103 and the timing mistakes of the musical instrument performance detected in step S104 (step S105).

[0032] For example, the performance scoring unit 15 displays on the screen whether the piece was "played", which part was wrong and how, advice on how to correct the mistake, recommendations on the next practice phrase to be practiced, and the like.

[0033] FIG. 3 is a diagram showing an example of musical score data of a model performance.

[0034] Instrument data of model performances that serve as models (correct answers) for instrument performances is stored in the music score data storage unit 12. In case an instrument performance close to the model performance is performed, the praise phrase "You played it!" is associated with the model performance and stored.

[0035] 4 and 5 are diagrams showing examples of musical score data of incorrectly performed pieces.

[0036] The score data of typical mistakes in musical instrument performance and correction advice phrases are stored in a score data storage unit 12 in association with each other.

[0037] Mistakes A to C shown in Fig. 4 are sheet music data of mistakes in constituent notes. A constituent note mistake is a mistake in which the piano key or the like is pressed incorrectly compared to the model performance. There are sections where the constituent notes are different from the constituent notes in the model performance, and mistakes in pitch, misalignment of chord timing, misalignment of left and right timing, etc. are included.

[0038] Mistakes D to F in Fig. 5 are sheet music data with timing mistakes. A timing mistake is a mistake in the timing of pressing or releasing a piano key, etc., compared to the model performance. The constituent notes are correct throughout the entire section, but there is some stretching and shrinking of the timing.

[0039] The praise phrases illustrated in FIG. 3 and the correction advice phrases illustrated in FIG. 4 and FIG. 5 are used when the performance sound scoring unit 16 displays the scoring results of the musical instrument performance.

[0040] We will provide additional information on how to create score data for model performances and score data for incorrect performances.

[0041] Although all the score data of the practice piece may be registered, it is preferable to break the practice piece into task elements. It is preferable to prepare many practice phrases for each task element, such as short phrases of about 4 to 8 bars, and for each detailed level of difficulty, so that the student can continuously feel a sense of accomplishment one by one, starting from the easy ones. There may be one or more types of incorrect performance. By generating such score data, the effectiveness of homework is improved, and the practice piece can be played efficiently.

[0042] FIG. 6 is a diagram showing a flow of the performance error detection process performed by the performance error detector 14.

[0043] The performance error detection unit 14 reads out the score data of the model performance and the score data of the erroneous performance from the score data storage unit 12, and for each of them, as shown in FIG. 7, mechanically generates an SMF (Standard MIDI File) from the score data, mechanically generates a WAV (Waveform Audio Format) from the SMF, and extracts features from the WAV to generate time-series feature data of the model performance model WAV and time-series feature data of the erroneous performance model WAV (step S201).

[0044] SMF is a file that records MIDI messages (such as key press: note ON, key release: note OFF, pitch: note number, strength: velocity) along with timing information. WAV is a music file format.

[0045] This extracts time-series feature data, as shown in the bottom row of Fig. 7. This feature data is generally called a spectrogram, and shows the time-series frequency components of the sound. The horizontal axis is time, and the vertical axis is pitch (a musical scale, frequency on a logarithmic axis).

[0046] In the example of Figure 7, the white parts represent the parts with strong frequency components, and the black parts represent the parts with weak frequency components. C4 is the middle C on the piano, C3 is the C one octave lower than C4 (its frequency is half that of C4), and C5 is the C one octave higher than C4 (its frequency is twice that of C4). From the feature data shown in Figure 7, it is possible to know when to strike and when to release which key.

[0047] 8 is a diagram showing the time-series feature amount data of mistake A (a mistake on a constituent note). The points indicated by the arrows are different from the feature amount data of the model performance shown in FIG.

[0048] The model performance WAV feature data and the incorrect performance WAV feature data can be easily created by an instructor with knowledge and experience of teaching music theory and piano. Literacy in statistics or information processing is not required. If an instructor creates these models, step S201 can be omitted.

[0049] Next, the performance error detection unit 14 mechanically generates a WAV file from the recording data of the musical instrument performance performed by the user, as shown in FIG. 9, and extracts features from the WAV file to generate time-series feature data of the musical instrument performance WAV (step S202).

[0050] The recording data of the musical instrument performance is recorded using a microphone on a smartphone, tablet terminal, etc. In this way, since the recording data of the musical instrument performance is used, the present disclosure can also be applied to acoustic instruments that do not support MIDI.

[0051] In addition, there are cases where privacy needs to be considered for recorded data of musical instrument performances, so when scoring is performed on the cloud, it is preferable to perform the process from recording to feature extraction locally, and send only feature data that is difficult to restore, such as voice, to the cloud for scoring.

[0052] Next, the performance error detection unit 14 calculates the DTW distance between the feature amount data of the musical instrument performance WAV and each of the feature amount data of the exemplary performance model WAV and each of the error performance model WAV (step S203).

[0053] Specifically, the DTW distance is calculated between the feature data of the musical instrument performance WAV and the feature data of the model performance WAV. Similarly, the DTW distance is calculated between the feature data of the musical instrument performance WAV and the feature data of the mistake performance model WAV (Miss A). The DTW distance is also calculated between the feature data of the musical instrument performance WAV for each of mistakes B to F.

[0054] DTW is a type of DP matching. It is used when measuring the similarity between time series data, such as template matching and pattern matching. Specifically, it is a method for finding the shortest path between two pieces of time series data, first and second time series data, by calculating all the distances between each point in the first time series data and each point in the second time series data. The distance of this shortest path is called the DTW distance.

[0055] For example, regarding the relationship between the feature data of the model performance model WAV and the feature data of the correct score performance WAV, a path like that shown in Figure 10 is calculated. The horizontal axis is the time series feature data of the correct score performance WAV, which is performed exactly according to the model performance score. The vertical axis is the time series feature data of the model performance model WAV. Although there is a difference in tempo between each note-ON time (keystroke time), they progress at the same rate, so the slope of the path passing through each keystroke time is constant and the distance is the shortest.

[0056] Next, the performance error detection unit 14 compares the calculated seven DTW distances with each other and determines the closest DTW distance, thereby determining the performance that is closest to the instrument performance from among the model performance and the incorrect performance (step S204).

[0057] Finally, the performance error detection unit 14 detects the presence or absence of an error in the musical instrument performance based on the determined performance (step S205).

[0058] Specifically, if the performance is closest to mistake A, it is detected that the musical instrument performance had a mistake in the same constituent notes as mistake A, and a correction advice phrase associated with the score data of mistake A is sent to the performance sound scoring unit 15. The same is done for the timing mistakes of mistakes D to F. If the performance is closest to the model performance, it is detected that there was no mistake, and a praise phrase associated with the score data of the model performance is sent to the performance sound scoring unit 15.

[0059] FIG. 11 is a diagram showing the result of error determination by the performance error detector 14. As shown in FIG.

[0060] Four types of instruments were used: a grand piano, an upright piano, an electronic piano (heavy keys), and a keyboard (light keys). For the score of Op.13248, a total of seven types of score data were used: the model performance score data and the score data for mistakes A to F. Performances of the four types of instruments were recorded simultaneously using multiple different types of smartphones and tablet devices. Figure 11 shows some of the evaluation results.

[0061] The correct judgment rate was high for mistakes A through C, which were mistakes in the constituent notes. On the other hand, the correct judgment rate was low for mistakes D through F, which were timing mistakes, and for cases where the correct score was played.

[0062] We will consider this point. It may be possible to determine timing errors from the slope of the path as shown in Figure 10. However, DTW allows for expansion and contraction or shifting of each point (note-on time) in two time series data along the time axis, so the positions of corresponding points in the time series data shift forward or backward, making it difficult to determine timing errors from the slope of the path. In other words, it is difficult to detect timing errors with DTW.

[0063] Therefore, for timing errors, a method is used to find the correspondence relationship by considering the waveform of the time-series data. Specifically, for example, the above-mentioned DDTW method, which is a method of differentiating the time-series data of the feature amount and then performing DTW, is used.

[0064] Moreover, performance errors are detected in two stages. Specifically, in the first stage, score data for errors A to C is used to detect component note errors using DTW. In the second stage, score data for errors D to F is used to detect timing errors using DDTW or DTW after differentiation. The second stage detection method takes into account the waveform of the time series data, so there is no shift in the positions of corresponding points in the time series data, and the corresponding keystroke times can be determined.

[0065] FIG. 12 is a diagram showing a flow of the performance error detection process performed by the performance error detector 14.

[0066] Step S205 shown in FIG. 6 is improved.

[0067] In the first stage of processing, the performance error detection unit 14 performs steps S201 to S204 using the score data of errors A to C, and determines whether the performance that is closest to the instrument performance is the model performance as a result of the determination in step S204 (step S301).

[0068] If the performance that is closest to the instrument performance is the model performance, the performance error detection unit 14 proceeds to the second stage of processing and detects timing errors using the score data of errors D to F (step S302).

[0069] If the performance closest to the instrument performance is not the model performance, the performance error detection unit 14 detects that there has been a constituent note error similar to the closest incorrect performance (any of the constituent note errors A to C), and ends the process (step S303).

[0070] FIG. 13 is a diagram showing a flow of the timing error detection process performed by the performance error detector 14.

[0071] This is a detailed description of step S302.

[0072] The performance error detection unit 14 extracts time-series sound pressure data from the exemplary performance model WAV (step S401).

[0073] Next, the performance error detection unit 14 extracts time-series sound pressure data from the musical instrument performance WAV (step S402).

[0074] Next, the performance error detection unit 14 compares the sound pressure data of the model performance WAV with the sound pressure data of the instrument performance WAV using the DDTW, detects the corresponding sound pressure peak time (keystroke time), and calculates the difference from the previous keystroke time (keystroke time interval) (step S403).

[0075] DDTW is a method that focuses on the degree of change between two time series data. It can detect the time of the corresponding point in the time series data. In other words, as shown in Figure 14, it can calculate which time in the musical instrument performance WAV the known keystroke time already added to the model performance WAV corresponds to, and how much it has expanded or contracted.

[0076] Next, the performance error detection unit 14 determines whether the time (keystroke time) of the peak detected sound pressure and the length (keystroke time interval) between the time and the previous keystroke time correspond to those represented in the erroneous performances (mistakes D to F) (step S404).

[0077] For example, as shown in Fig. 15, for the sheet music data of an incorrect performance, the keying positions where there are errors and the keying time intervals between them are preset in advance compared to the sheet music data of a model performance. Then, it is determined which combination of keying positions and keying time interval expansion / contraction thresholds set for mistakes D to F corresponds to the keying time and keying time interval detected in step S403.

[0078] At this time, the difference in tempo between the instrument performance WAV and the model performance WAV may be corrected. The tempo of each performance is assumed to be constant, and the difference between the last keystroke time and the first keystroke time is calculated, and the keystroke time interval is corrected using the ratio as the tempo ratio before being compared with the threshold value.

[0079] If the mistake corresponds to any of the score data for the incorrect performance (mistakes D to F), the performance error detection unit 14 detects that a corresponding timing mistake has occurred, and passes the correction advice phrase (Figure 5) associated with the corresponding incorrect performance to the performance sound scoring unit 15 (step S405).

[0080] For example, when the judgment condition for mistake F is satisfied, it is detected that the musical instrument performance has a timing mistake similar to mistake F, and the advice phrase associated with mistake F is passed to the performance sound scoring unit 15 .

[0081] If the performance does not match any of the score data for the incorrect performance (mistakes D to F), that is, if none of the judgment conditions for mistakes D to F are met, the performance error detection unit 14 detects that there was no timing error, and passes the praise phrase (Figure 3) associated with the model performance to the performance sound scoring unit 15 (step S406).

[0082] Thereafter, the performance sound scoring section 15 scores the performance sound of the musical instrument performance based on the detection result of the presence or absence of component sound mistakes or timing mistakes, displays the scoring result on the screen, and displays a praise phrase or a correction advice phrase on the screen.

[0083] 16 is a diagram showing the results of performance error determination by the performance error detection unit 14. It can be seen that the rate of correct determination of correct score performance and timing errors of errors D to F is improved compared to the determination results shown in FIG.

[0084] [Variations] In this embodiment, the component sound error is detected first, but the timing error may be detected first, or the timing error and the component sound error may be detected simultaneously.

[0085] [effect] According to this embodiment, the performance error detection unit 14 extracts time-series first feature amount data representing the feature amount of the musical instrument performance and time-series second feature amount data representing the feature amounts of the model performance and the incorrect performance, compares the magnitude of each distance between the first feature amount data and each second feature amount data using AI that measures the similarity between the time-series data, and detects the model performance or the incorrect performance related to the second feature amount data that is closest to the first feature amount data, thereby detecting the presence or absence of a component note error in the musical instrument performance and, if there is a component note error, the position of the component note error, and extracts the time-series First sound pressure data and second sound pressure data that is a time series of an exemplary performance are extracted, and the AI ​​that measures the similarity between the time series data taking into account the shape of the waveforms is used to detect each position of sound pressure that corresponds between the first sound pressure data and the second sound pressure data and the time difference between those positions. It is then determined whether the detected positions and time differences in the musical instrument performance correspond to the positions and time differences in an incorrect performance, thereby detecting whether there is a timing mistake in the musical instrument performance, and if there is a timing mistake, the position of the timing mistake is detected, thereby making it possible to detect both mistakes in the constituent notes and timing mistakes in the musical instrument performance with a high degree of accuracy.

[0086] Furthermore, according to this embodiment, mistakes in musical instrument performance are detected using recorded data of the musical instrument performance, so mistakes in the performance of acoustic musical instruments, not electronic musical instruments, can be detected.

[0087] Furthermore, according to this embodiment, the performance error detection unit 14 detects constituent note errors using only the score data of constituent note errors, and detects timing errors using only the score data of timing errors, so that both constituent note errors and timing errors in musical instrument performance can be detected with higher accuracy.

[0088] [others] The present disclosure is not limited to the above-described embodiment, and various modifications are possible within the scope of the present disclosure.

[0089] The performance sound scoring device 1 of the present embodiment described above can be realized, for example, as shown in Fig. 17, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906. The memory 902 and the storage 903 are storage devices. In the computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of the performance sound scoring device 1.

[0090] The performance sound scoring device 1 may be implemented in one computer. The performance sound scoring device 1 may be implemented in multiple computers. The performance sound scoring device 1 may be a virtual machine implemented in a computer. The program for the performance sound scoring device 1 may be stored in a computer-readable recording medium such as a HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the performance sound scoring device 1 may also be distributed via a communication network.

[0091] [Note] This application discloses the following inventions.

[0092] (Appendix 1) In a musical performance assistance device for assisting musical instrument playing, a registration unit for registering each piece of score data of the model performance and the mistake performance in a storage unit; a reception unit for receiving recording data of musical instrument performance; a detection unit that detects mistakes in the musical instrument performance by using the score data of the model performance and the mistaken performance and the recording data of the musical instrument performance, The detection unit is extracting first feature amount data in a time series representing a feature of the musical instrument performance and each of second feature amount data in a time series representing each of the feature amounts of the model performance and the incorrect performance, comparing the magnitude of each distance between the first feature amount data and each of the second feature amount data using an AI that measures the similarity between the time series data, and detecting the model performance or the incorrect performance related to the second feature amount data that is closest to the first feature amount data, thereby detecting whether or not there is a mistake in a constituent note in the musical instrument performance and, if there is a mistake in a constituent note, the position of the mistake in a constituent note; The performance assistance device extracts first time-series sound pressure data of the musical instrument performance and second time-series sound pressure data of the model performance, detects corresponding sound pressure positions and time differences between positions between the first sound pressure data and the second sound pressure data using an AI that measures the similarity between the time-series data taking into account the shape of the waveforms, and determines whether the detected positions and time differences in the musical instrument performance correspond to positions and time differences in the incorrect performance, thereby detecting whether or not there is a timing error in the musical instrument performance and, if there is a timing error, the position of the timing error.

[0093] (Appendix 2) The detection unit is 2. The assistive playing device according to claim 1, which detects constituent note errors using only musical score data of constituent note errors, and detects timing errors using only musical score data of timing errors.

[0094] (Appendix 3) The AI ​​that measures the similarity between the time series data is: DTW (Dynamic Time Warping) The AI ​​measures the similarity between time series data taking into account the shape of the waveforms. Assistive playing device according to claim 1, which is a Derivative DTW (DDTW). [Explanation of symbols]

[0095] 1. Performance sound scoring device (including performance assistance device) 11 Music score data registration section (registration section) 12 Music score data storage unit (storage unit) 13 Recording Data Reception Section (Reception Section) 14 Performance error detection unit (detection unit) 15 Performance Scoring Department 901:CPU 902: Memory 903: Storage 904: Communication equipment 905: Input device 906: Output device

Claims

1. In a musical performance assistance device for assisting musical instrument playing, a registration unit for registering each piece of score data of the model performance and the mistake performance in a storage unit; a reception unit for receiving recording data of musical instrument performance; a detection unit that detects mistakes in the musical instrument performance by using the score data of the model performance and the mistaken performance and the recording data of the musical instrument performance, The detection unit is extracting time-series first feature amount data representing a feature of the musical instrument performance and time-series second feature amount data representing each feature of the model performance and the incorrect performance, comparing the magnitude of each distance between the first feature amount data and each of the second feature amount data using AI that measures the similarity between the time-series data, and detecting the model performance or the incorrect performance related to the second feature amount data that is closest to the first feature amount data, thereby detecting whether or not there is a mistake in a constituent note in the musical instrument performance and, if there is a mistake in a constituent note, the position of the mistake in a constituent note; The performance assistance device extracts first time-series sound pressure data of the musical instrument performance and second time-series sound pressure data of the model performance, detects each position of sound pressure that corresponds to the first sound pressure data and the second sound pressure data and the time difference between the positions using an AI that measures the similarity between the time-series data taking into account the shape of the waveform, and determines whether the detected positions and time differences in the musical instrument performance correspond to the positions and time differences in the incorrect performance, thereby detecting whether or not there is a timing error in the musical instrument performance and, if there is a timing error, the position of the timing error.

2. The detection unit is 2. The assistive playing device according to claim 1, wherein the musical score data for component note mistakes is used to detect the component note mistakes, and the musical score data for timing mistakes is used to detect the timing mistakes.

3. The AI ​​that measures the similarity between the time series data is: DTW (Dynamic Time Warping) The AI ​​measures the similarity between time series data taking into account the shape of the waveforms.

2. The assistive playing device according to claim 1, which is a Derivative DTW (DDTW) device.

4. A method for assisting a person in playing a musical instrument, comprising: The performance assistance device is A first step of registering each piece of music score data of the model performance and the mistake performance in a storage unit; A second step of accepting a recording of a musical instrument performance; a third step of detecting mistakes in the musical instrument performance using AI, using the score data of the model performance and the mistaken performance and the recording data of the musical instrument performance; In the third step, extracting time-series first feature amount data representing a feature of the musical instrument performance and time-series second feature amount data representing each feature of the model performance and the incorrect performance, comparing the magnitude of each distance between the first feature amount data and each of the second feature amount data using AI that measures the similarity between the time-series data, and detecting the model performance or the incorrect performance related to the second feature amount data that is closest to the first feature amount data, thereby detecting whether or not there is a mistake in a constituent note in the musical instrument performance and, if there is a mistake in a constituent note, the position of the mistake in a constituent note; A performance assistance method that detects whether or not there is a timing error in the musical instrument performance and, if there is a timing error, the position of the timing error, by extracting first time-series sound pressure data of the musical instrument performance and second time-series sound pressure data of the model performance, detecting each position of sound pressure that corresponds between the first sound pressure data and the second sound pressure data and the time difference between the positions using an AI that measures the similarity between the time-series data taking into account the shape of the waveforms, and determining whether the detected positions and time differences in the musical instrument performance correspond to the positions and time differences in the incorrect performance.

5. 2. An assistive music playing program for causing a computer to function as the assistive music playing device according to claim 1.

Images