Fingering practice device, fingering practice method, and training method

Abstract

This fingering presentation device comprises a reception unit and an estimation unit. The reception unit receives time-series data including a note sequence consisting of a plurality of notes. The estimation unit estimates finger information by using a trained model. The finger information indicates fingers to be used when playing, with the musical instrument, at least a part of the notes included in the note sequence received by the reception unit. Alternatively, the estimation unit estimates note information by using the trained model. The note information indicates a note to which a fingering is to be given, in the note sequence received by the reception unit.

Description

[Technical field]

[0001] The present invention relates to a fingering technique providing device, a fingering technique providing method, and a fingering technique training method for providing fingering techniques for playing a musical instrument. [Background technology]

[0002] There are known devices for assisting practice of playing musical instruments. For example, in an information processing device described in Patent Document 1, a performance technical level of a player is calculated, and musical pieces that the player can play are presented based on the calculated performance technical level. However, if the player is an inexperienced player, it is not easy to appropriately determine fingering (hereinafter referred to as fingering) for playing each note on an instrument. In response to this, Patent Document 2 describes a fingering determination method that determines fingering for each note of a note sequence based on a probabilistic model. [Patent Document 1] JP 2013-083845 A [Patent Document 2] JP 2007-241034 A Summary of the Invention [Problem to be solved by the invention]

[0003] According to Patent Document 2, a player can recognize fingerings for playing musical instruments based on a probabilistic model. However, in reality, there are an infinite number of fingering combinations, and there is not one optimal fingering for playing a piece of music. Therefore, it is desirable to present more appropriate fingerings.

[0004] An object of the present invention is to provide a fingering practice device, a fingering practice method, and a fingering practice training method that are capable of presenting appropriate fingering when playing a musical instrument. [Means for solving the problem]

[0005] The present invention 1.The fingering guidance presentation device according to the present invention includes a reception unit that receives time-series data including a sequence of notes including a plurality of notes, and a trained model that receives fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument. of and an estimation unit that estimates fingering information based on the player identifier, the time series data further including a player identifier that indicates a player who plays the sequence of notes. The trained model is a machine learning model that has acquired an input / output relationship between input time-series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including a reference performer identifier indicating a reference performer who plays the reference note sequence, and the output fingering information indicating the fingers of the reference performer to be used when playing the notes on a musical instrument. . The present invention Second The fingering technique presentation device according to the present invention includes a reception unit that receives time-series data including a sequence of notes, and a trained model that receives fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument. of and an estimation unit that estimates, based on the basic fingering information, fingering information indicating fingers to be used when playing, on a musical instrument, notes in a first proportion of the notes included in the sequence of notes, the second proportion being greater than the first proportion. The trained model is a machine learning model that has acquired an input / output relationship between input time-series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including basic fingering information indicating fingers to be used when playing a first proportion of the notes included in the reference note sequence on a musical instrument, and the output fingering information further including the basic fingering information included in the input time-series data. . The present invention Third The fingering suggestion device according to the above aspect includes: a receiving unit that receives time-series data including a sequence of notes that includes a plurality of notes; and, using a trained model, fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument. and and an estimation unit that estimates note information indicating a note to which fingering is to be assigned from the sequence of notes, the estimation unit estimating intermediate fingering information indicating a finger to be used when playing each note included in the sequence of notes with a musical instrument and the note information, and by deleting fingering information corresponding to the note to which fingering is to be assigned in the note information from Then, fingering information indicating the fingers to be used when playing notes included in the sequence of notes, other than the notes indicated by the note information, on an instrument, is estimated.

[0006] The present invention FourthThe training device according to the above aspect includes a first acquisition unit that acquires input time series data including a reference note sequence consisting of a plurality of notes, a second acquisition unit that acquires output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on an instrument, or output note information indicating notes to which fingering is to be assigned from the reference note sequence, and a construction unit that constructs a trained model that has acquired an input / output relationship between the input time series data and the output fingering information or the output note information, wherein the input time series data further includes a reference player identifier indicating a reference player playing the reference note sequence, and the output fingering information indicates the fingers of the reference player to be used when playing on the instrument.

[0007] The present invention Fifth The fingering suggestion method according to the aspect of the present invention receives time-series data including a sequence of notes, and uses a trained model to provide fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument. of the time-series data further includes a performer identifier indicating a performer performing the sequence of notes, and estimating finger information includes estimating finger information based on the performer identifier; The trained model is a machine learning model that has acquired an input / output relationship between input time-series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including a reference performer identifier indicating a reference performer who plays the reference note sequence, the output fingering information indicating the fingers of the reference performer to be used when playing the notes on a musical instrument, It is executed by a computer. The present invention Sixth The fingering suggestion method according to the aspect of the present invention receives time-series data including a sequence of notes, and uses a trained model to provide fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument. of the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of notes included in the sequence of notes on a musical instrument, and estimating the fingering information includes estimating fingering information indicating fingers to be used when playing a second proportion of notes included in the sequence of notes on a musical instrument, the second proportion being greater than the first proportion, based on the basic fingering information; The trained model is a machine learning model that has acquired an input / output relationship between input time-series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including basic fingering information indicating fingers to be used when playing a first proportion of the notes included in the reference note sequence on a musical instrument, and the output fingering information further including the basic fingering information included in the input time-series data, It is executed by a computer. The present invention Seventh The fingering suggestion method according to the aspect of the present invention includes receiving time-series data including a sequence of notes, and using a trained model, providing fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument; andThe estimation of note information indicating the notes to which fingering is to be applied from the sequence of notes and the estimation of fingering information include estimating intermediate fingering information indicating the fingers to be used when playing each note included in the sequence of notes with an instrument and the note information, and estimating the intermediate fingering information. by deleting fingering information corresponding to the note to which fingering is to be assigned in the note information from The method includes estimating finger information indicating fingers to be used when playing notes included in the sequence of notes, other than the notes indicated by the note information, on an instrument, and is executed by a computer.

[0008] The present invention Eighth A training method according to the above aspect includes: acquiring input time series data including a reference note sequence consisting of a plurality of notes; acquiring output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on an instrument, or output note information indicating notes to which fingering is to be assigned from the reference note sequence; and constructing a trained model that has acquired an input / output relationship between the input time series data and the output fingering information or the output note information, wherein the input time series data further includes a reference player identifier indicating a reference player playing the reference note sequence, and the output fingering information indicates the fingers of the reference player to be used when playing on the instrument, and is executed by a computer. Effect of the Invention

[0009] According to the present invention, it is possible to present appropriate fingering when playing a musical instrument. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of a processing system including a fingering technique providing device and a training device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of each training data. [Diagram 3] FIG. 3 is a block diagram showing the configuration of the training device and the fingering technique presentation device. [Figure 4] FIG. 4 shows an example of the auxiliary musical score displayed on the display unit. [Diagram 5] FIG. 5 is a flowchart showing an example of a training process performed by the training device of FIG. [Figure 6]FIG. 6 is a flowchart showing an example of a fingering practice process performed by the fingering practice device of FIG. [Figure 7] FIG. 7 is a diagram showing another example of the input time series data. [Figure 8] FIG. 8 is a diagram showing an example of input time-series data in the modified example. [Figure 9] FIG. 9 is a diagram showing an example of output finger information in the modified example. [Figure 10] FIG. 10 is a diagram illustrating an example of input time-series data in the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of output finger information in the third embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of a fingering guide suggestion process in the modified example. [Figure 13] FIG. 13 is a diagram showing an example of finger information estimated in step S24 of the fingering suggestion process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [1] First embodiment (1) Processing system configuration Hereinafter, a fingering practice device, a training device, a fingering practice method, and a training method according to embodiments of the present invention will be described in detail with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a fingering practice device and a training device according to a first embodiment of the present invention. As shown in Fig. 1, the processing system 100 includes a RAM (random access memory) 110, a ROM (read only memory) 120, a CPU (central processing unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.

[0012] The processing system 100 is realized by a computer such as a personal computer, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet, or may be realized by an electronic musical instrument with a performance function such as an electronic piano.

[0013] The RAM 110, the ROM 120, the CPU 130, the storage unit 140, the operation unit 150 and the display unit 160 are connected to a bus 170. The RAM 110, the ROM 120 and the CPU 130 constitute the training device 10 and the fingering presentation device 20. In this embodiment, the training device 10 and the fingering presentation device 20 are constituted by a common processing system 100, but may be constituted by separate processing systems.

[0014] The RAM 110 is, for example, a volatile memory, and is used as a working area for the CPU 130. The ROM 120 is, for example, a non-volatile memory, and stores a training program and a fingering pattern presentation program. The CPU 130 performs training processing by executing the training program stored in the ROM 120 on the RAM 110. The CPU 130 also performs fingering pattern presentation processing by executing the fingering pattern presentation program stored in the ROM 120 on the RAM 110. The training processing and the fingering pattern presentation processing will be described in detail later.

[0015] The training program or the fingering technique presentation program may be stored in the storage unit 140 instead of the ROM 120. Alternatively, the training program or the fingering technique presentation program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the training program or the fingering technique presentation program may be distributed from a server (including a cloud server) on the network and installed in the ROM 120 or the storage unit 140.

[0016] The storage unit 140 includes a storage medium such as a hard disk, an optical disk, a magnetic disk, or a memory card, and stores the trained model M and a plurality of training data D. The trained model M or each of the training data D may be stored in a computer-readable storage medium instead of being stored in the storage unit 140. Alternatively, when the processing system 100 is connected to a network, the trained model M or each of the training data D may be stored in a server on the network.

[0017] (2) Training data The trained model M is a machine learning model trained to present fingering to a user (hereinafter referred to as a performer) of the fingering presentation device 20 when playing a musical piece on an instrument, and is constructed using a plurality of training data D. A user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is data created based on the performance knowledge or playing style of a reference performer. The reference performer has a relatively high level of skill in playing a musical piece. The reference performer may be a coach or teacher of the performer in playing the musical piece.

[0018] The training data D indicates a set of input time series data and output fingering information. The input time series data indicates a reference note sequence consisting of a plurality of notes. The input time series data may be image data showing an image of a musical score. The output fingering information indicates the fingers of the reference performer to be used when playing each note of the reference note sequence with an instrument, and may be used to present fingering when playing the reference note sequence. The output fingering information may be a unique number assigned to each finger. In this example, the numbers "1" to "5" are assigned to the thumb, index finger, middle finger, ring finger, and little finger, respectively.

[0019] Here, the optimal fingering for playing a piece of music differs depending on the physical characteristics of the performer or the performance style of the performer. Therefore, in this embodiment, the input time-series data further includes a reference performer identifier indicating a classification (category) of the reference performer who performs the reference sequence of notes. The reference performer identifier is determined to be different for at least one of the physical characteristics of the reference performer and the performance style of the reference performer. The physical characteristics of the reference performer include, for example, the hand size (finger length), age, sex, or whether the reference performer is an adult or a child.

[0020] FIG. 2 is a diagram showing an example of each training data D. The example in FIG. 2 shows a part of the input time series data and the output fingering information when a reference performer plays the piano. As shown in FIG. 2, the input time series data A includes elements A0 to A16. The element A0 corresponds to a reference performer identifier and is represented by a character string that differs for at least one of the physical characteristics of the reference performer and the style of performance by the reference performer. The elements A1 to A16 correspond to the reference note sequence. In this example, the element A0 is placed at the beginning of the input time series data A, that is, before the reference note sequence (elements A1 to A16), but may be placed at any position in the input time series data A.

[0021] In elements A1, A3, A5, ..., A15, "L" means the left hand, the numbers mean the numbers assigned to the keys, and "on" and "off" mean pressing and releasing the keys, respectively. In elements A2, A4, A6, ..., A16, "wait" means waiting, and the numbers mean the length of time. Therefore, elements A1 to A4 mean pressing key "66" and holding it for 13 units of time, and then releasing key "66" and holding it for 2 units of time.

[0022] The output fingering information B includes elements B0 to B16 corresponding to the elements A0 to A16 of the input time-series data A, respectively. The element B0 indicates a reference player identifier and is represented by the same character string as the element A0. In the elements B1, B3, B5, ..., B15, "L" means the left hand, the numbers mean the numbers given to the fingers, and "down" and "up" mean pushing up and pushing down, respectively. In the elements B2, B4, B6, ..., B16, "wait" means waiting, and the numbers mean the length of time. Therefore, the elements B1 to B4 mean to push down the middle finger of the left hand and wait for 13 units of time, and then to hold the middle finger of the left hand up for 2 units of time.

[0023] 2 is generated to indicate fingering for the left hand, the embodiment is not limited thereto. The training data D may be generated to indicate fingering for the right hand, or may be generated to indicate fingering for each of the left and right hands. In the elements of the input time-series data A and the output fingering information B to indicate fingering for the right hand, the letter "R", for example, may be used instead of "L".

[0024] (3) Training device and fingering guidance device Fig. 3 is a block diagram showing the configuration of the training device 10 and the fingering presentation device 20. As shown in Fig. 3, the training device 10 includes, as functional units, a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13. The functional units of the training device 10 are realized by the CPU 130 in Fig. 1 executing a training program. At least a part of the functional units of the training device 10 may be realized by hardware such as an electronic circuit.

[0025] The first acquisition unit 11 acquires input time series data A from each training data D stored in the storage unit 140 or the like. The second acquisition unit 12 acquires output finger information B from each training data D. The construction unit 13 performs machine learning for each training data D, with the input time series data A acquired by the first acquisition unit 11 as an input element and the output finger information B acquired by the second acquisition unit 12 as an output element. By repeating machine learning for multiple training data D, the construction unit 13 constructs a trained model M indicating an input / output relationship between the input time series data A and the output finger information B.

[0026] In this example, the construction unit 13 constructs the trained model M by training a Transformer, but the embodiment is not limited to this. The construction unit 13 may construct the trained model M by training another type of machine learning model that handles time series. The trained model M constructed by the construction unit 13 is stored in, for example, the storage unit 140. The trained model M constructed by the construction unit 13 may be stored in a server on a network, etc.

[0027] The fingering presentation device 20 includes, as functional units, a receiving unit 21, an estimating unit 22, and a generating unit 23. The CPU 130 in Fig. 1 executes a fingering presentation program to realize the functional units of the fingering presentation device 20. At least a part of the functional units of the fingering presentation device 20 may be realized by hardware such as an electronic circuit.

[0028] In this embodiment, the receiving unit 21 receives time-series data including a string of notes consisting of a plurality of notes. The performer can provide image data showing an image of a musical score to the receiving unit 21 as time-series data. Alternatively, the performer can generate time-series data by operating the operation unit 150 and provide the time-series data to the receiving unit 21.

[0029] In this example, the time series data has a structure similar to that of the input time series data A in FIG. 2, and further includes a performer identifier indicating a category of a performer who performs the sequence of notes. The performer identifier is determined to be different for at least one of the physical characteristics of the performer and the performance style of the performer. The physical characteristics of the performer include, for example, the size of the performer's hands, age, sex, or whether the performer is an adult or a child.

[0030] The estimation unit 22 estimates finger information using a trained model M stored in the storage unit 140 or the like. The finger information indicates the player's fingers used when playing each note of the note sequence accepted by the acceptance unit 21, and is estimated based on the note sequence and a player identifier. The finger information may be a unique number assigned to each finger. The generation unit 23 generates musical score information based on the note sequence of the time-series data accepted by the acceptance unit 21 and the finger information estimated by the estimation unit 22.

[0031] The auxiliary score is displayed on the display unit 160 based on the score information generated by the generation unit 23. Fig. 4 shows an example of the auxiliary score displayed on the display unit 160. As shown in Fig. 4, the auxiliary score shows finger information estimated by the estimation unit 22 corresponding to each note of the sequence of notes accepted by the acceptance unit 21. In the example of Fig. 4, the finger numbers of one hand are shown as the finger information.

[0032] When distinguishing between the numbers of fingers on the left and right hands, a predetermined letter such as "L" may be added near the number of the finger on the left hand, and another predetermined letter such as "R" may be added near the number of the finger on the right hand. Alternatively, a predetermined color such as red may be added to the numbers of the fingers on the left hand or the corresponding notes, and another predetermined color such as blue may be added to the numbers of the fingers on the right hand or the corresponding notes.

[0033] (4) Training and fingering display processing Fig. 5 is a flowchart showing an example of training processing by the training device 10 of Fig. 3. The training processing of Fig. 5 is performed by the CPU 130 of Fig. 1 executing a training program. First, the first acquisition unit 11 acquires input time-series data A from each training data D (step S1). Furthermore, the second acquisition unit 12 acquires output finger information B from each training data D (step S2). Either step S1 or S2 may be executed first, or steps S1 and S2 may be executed simultaneously.

[0034] Next, the construction unit 13 performs machine learning for each training data D using the input time-series data A acquired in step S1 as an input element and the output finger information B acquired in step S2 as an output element (step S3). Next, the construction unit 13 determines whether or not sufficient machine learning has been performed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeated while changing the parameters until sufficient machine learning has been performed. The number of times the machine learning is repeated changes depending on the quality conditions that the trained model M to be constructed should satisfy.

[0035] When sufficient machine learning has been performed, the construction unit 13 stores the input / output relationship between the input time-series data A and the output finger information B acquired by the machine learning in step S3 as a trained model M (step S5). This ends the training process.

[0036] Fig. 6 is a flowchart showing an example of fingering presentation processing by the fingering presentation device 20 of Fig. 3. The fingering presentation processing of Fig. 6 is performed by the CPU 130 of Fig. 1 executing a fingering presentation program. First, the receiving unit 21 receives time-series data (step S11). Next, the estimation unit 22 estimates finger information from the time-series data received in step S11 by using the trained model M stored in step S5 of the training processing (step S12).

[0037] After that, the generation unit 23 generates score information based on the sequence of notes in the time-series data accepted in step S11 and the fingering information estimated in step S12 (step S13). Based on the generated score information, an auxiliary score may be displayed on the display unit 160. This ends the fingering suggestion process.

[0038] (5) Effects of the embodiment As described above, the fingering presentation device 20 according to this embodiment includes a receiving unit 21 that receives time-series data including a note sequence consisting of a plurality of notes, and an estimating unit 22 that estimates fingering information indicating the finger to be used when playing each note of the note sequence on an instrument, using a trained model M. According to this configuration, appropriate fingering information is estimated from the time flow of the plurality of notes in the time-series data, using the trained model M. This makes it possible to present appropriate fingering when playing an instrument.

[0039] The trained model M may be a machine learning model that has learned the input / output relationship between input time-series data A including a reference sequence of notes, and output fingering information B indicating the finger to be used when playing each note of the reference sequence of notes on an instrument. In this case, the fingering information can be easily estimated from the time-series data.

[0040] The time-series data may further include a performer identifier that indicates a performer who performs the sequence of notes, and the estimation unit 22 may estimate fingering information based on the performer identifier. In this case, it is possible to estimate appropriate fingering information according to the performer.

[0041] The performer identifier may be determined to correspond to the physical characteristics of the performer, in which case appropriate fingering information can be estimated according to the physical characteristics of the performer.

[0042] The performer identifier may be determined so as to correspond to the performance style of the performer. In this case, appropriate fingering information can be estimated according to the performance style of the performer.

[0043] The fingering suggestion device 20 includes a generating unit that generates musical score information indicating an auxiliary musical score to which fingering information is added so as to correspond to each note of a musical note sequence. 23 In this case, the performer can easily recognize the finger that corresponds to each note of the sequence of notes by visually checking the auxiliary musical score.

[0044] The training device 10 according to this embodiment includes a first acquisition unit 11 that acquires input time series data A including a reference note sequence consisting of a plurality of notes, a second acquisition unit 12 that acquires output fingering information B indicating the finger to be used when playing each note of the reference note sequence on an instrument, and a construction unit 13 that constructs a trained model M that has learned the input / output relationship between the input time series data A and the output fingering information B. With this configuration, the trained model M that has learned the input / output relationship between the input time series data A and the output fingering information B can be easily constructed.

[0045] (6) Other examples of training data In this embodiment, the input time series data A includes a reference performer identifier, and the time series data includes a performer identifier, but the embodiment is not limited to this. The input time series data A only needs to include a reference note sequence, and does not need to include a reference performer identifier. Similarly, the time series data only needs to include a note sequence, and does not need to include a performer identifier.

[0046] In the present embodiment, the input time series data A and the output fingering information B are described in a so-called action-based manner, which indicates key pressing or key release in the MIDI (Musical Instrument Digital Interface) standard, but the present embodiment is not limited to this. The input time series data A and the output fingering information B may be described in other formats. For example, the input time series data A and the output fingering information B may be described in a so-called note-based manner, which indicates the start position or length of a note in the MIDI standard. The same applies to the time series data and fingering information.

[0047] FIG. 7 is a diagram showing another example of input time series data A. The upper part of FIG. 7 shows input time series data A (Ax) described on an action basis. The middle part of FIG. 7 shows input time series data A (Ay) described on a note basis. The input time series data Ax and the input time series data Ay contain the same reference note sequence (the reference note sequence in the musical score shown in the lower part of FIG. 7). "Bar" and "beat" in the input time series data Ax and Ay are elements that indicate the metrical structure of the reference note sequence.

[0048] As shown in Fig. 7, by describing the input time series data A on a note-by-note basis, the length of the input time series data A is shortened. This makes it easier to process longer input time series data A. Note that the output fingering information B corresponding to the input time series data A can be described by inserting an element indicating the finger number immediately after the element indicating the pitch number in the input time series data A ("note_○○").

[0049] Alternatively, the input time series data A and the output fingering information B may be described in a format that represents a musical score. Details of the input time series data A and the output fingering information B described in a format that represents a musical score will be described in the following modified example.

[0050] (7) Variations Fig. 8 is a diagram showing an example of input time series data A in the modified example. The upper part of Fig. 8 shows input time series data A (Az) described in a format that expresses musical scores. The lower part of Fig. 8 shows input time series data Az 8, the input time series data Az includes a plurality of elements A0 to A24. Some of the elements have attributes. The attributes of an element are written at the end of the element (after the underscore).

[0051] The element A0 indicates the ratio of the notes to which fingering is to be added among the notes included in the reference note sequence. The element A0 is the first note in the input time series data Az. ToHowever, it may be placed at any position in the input time series data Az. The percentage is specified by the attribute "fingerrate" in element A0. In this example, the attribute "5" means a percentage of 100%. The percentage may have a range, for example, 20 to 40% or 40 to 60%, or may be divided into multiple ranges.

[0052] Element A1 indicates a part. Element A1 is placed immediately after element A0, but may be placed at any position in the input time series data Az. In element A1, "R" and "L" indicate the right hand and left hand parts, respectively. In this example, the element corresponding to the right hand is placed after "R". Then "L" is placed, and the element corresponding to the left hand is placed after "L". "R" and the element corresponding to the right hand may be placed after the element corresponding to the left hand. When there is no distinction between parts, the input time series data Az does not include element A1.

[0053] Elements A2, A15, and A24 indicate bar lines in the musical score. Therefore, in the example of Fig. 8, the range delimited by "bar" in element A2 and "bar" in element A15 corresponds to the first bar. The range delimited by "bar" in element A15 and "bar" in element A24 corresponds to the second bar.

[0054] Element A3 indicates a clef in a musical score. The type of clef is specified by the "clef" attribute in element A3. In the example in Fig. 8, since the attribute is "treble", element A3 specifies a treble clef as the clef. If the attribute were "bass", element A3 would specify a bass clef as the clef.

[0055] Element A4 indicates the time signature of the musical score. The type of time signature is specified by the "time" attribute of element A4. In the example of Fig. 8, since the attribute is "4 / 4", element A4 specifies "4 / 4" as the time signature.

[0056] The notes in the reference note sequence are indicated by a pair of pitch and value. The pitch is specified by the "note" attribute in elements A5, A9, A11, A13, A16, A18, and A20. The value is specified by the "len" attribute in elements A6, A10, A12, A14, A17, A19, and A21. In this example, "len_1" corresponds to one beat.

[0057] The direction of the stem of the note in the musical score is specified by other attributes of "len" in elements A6, A10, A12, A14, A17, A19, and A21. If the other attribute is "down", the stem extends downward from the note head. If the other attribute is "up", the stem extends upward from the note head. When multiple notes such as eighth notes or sixteenth notes are beamed, the start, intermediate, and end positions of the beam are specified by further attributes of "len" in elements A10, A12, and A14, "start", "continue", and "stop", respectively.

[0058] A rest in the reference sequence is specified by the "rest" attribute in elements A7 and A22. The note value of the rest is described by the "len" attribute in elements A8 and A23.

[0059] In the example of Fig. 8, elements A5 and A6 indicate note N1, and elements A7 and A8 indicate rest R1. Elements A9 and A10 indicate note N2, elements A11 and A12 indicate note N3, and elements A13 and A14 indicate note N4. Elements A16 and A17 indicate note N5, and elements A18 and A19 indicate note N6. Elements A20 and A21 indicate note N7, and elements A22 and A23 indicate rest R2.

[0060] Fig. 9 is a diagram showing an example of output fingering information B in a modified example. The upper part of Fig. 9 shows output fingering information B (Bz) described in a format for expressing musical scores. The output fingering information Bz corresponds to the input time-series data Az in Fig. 8. The lower part of Fig. 9 shows the musical score represented by the output fingering information Bz.

[0061] As shown in the upper part of Fig. 9, the output finger information Bz includes a plurality of elements B0 to B24. The output finger information Bz further includes elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f arranged immediately after the elements B5, B9, B11, B13, B16, B18, and B20, respectively. The elements B0 to B24 are similar to the elements A0 to A24 of the input time series data Az in Fig. 8. Therefore, the first acquisition unit 11 in Fig. 3 can acquire the input time series data Az by deleting the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz.

[0062] The elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f indicate the finger numbers to be used when playing the notes corresponding to the immediately preceding elements B5, B9, B11, B13, B16, B18, and B20 on an instrument, respectively. The finger numbers are specified by the "finger" attribute in the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. Therefore, the finger numbers "1", "1", "2", "1", "3", "3", and "2" to be used when playing the notes N1 to N7 are written on the musical score by the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f, respectively, as shown in the lower part of FIG.

[0063] (8) Effects of Modifications In a modification of the first embodiment, the percentage of notes to which fingering is to be assigned among the notes included in the reference sequence of notes can be arbitrarily specified by the attribute of element A0. When the percentage is 100%, the estimation unit 22 estimates fingering information for all notes included in the sequence of notes. In this case, it is possible to present fingering appropriate for beginner-level players when playing musical instruments.

[0064] Also, when the ratio is 100%, the generating unit 23 may generate a moving image file showing the finger movement by animation or the like based on the finger information estimated by the estimating unit 22. This makes it possible to visualize the finger movement. The generation of such a moving image file may be executed before or after step S13 in the fingering guidance presentation process of FIG. 6, may be executed in parallel with step S13, or may be executed instead of step S13.

[0065] On the other hand, when the ratio is less than 100%, the estimation unit 22 estimates some of the notes included in the sequence of notes to which fingerings are to be assigned and fingering information for the some of the notes. In this case, it is possible to present appropriate fingerings for a player at a beginner level or intermediate level higher than the beginner level when playing an instrument. In this configuration, the output fingering information Bz does not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f.

[0066] On the other hand, when the ratio is less than 100%, the estimation unit 22 may estimate note information indicating the note to which fingering is to be assigned from the sequence of notes without estimating fingering information. Details will be described later in the third embodiment.

[0067] [2] Second embodiment (1) Processing system The processing system 100 in the second embodiment will be described with respect to differences from the processing system 100 in the first embodiment. In the training device 10 in Fig. 3, a first acquisition unit 11 and a second acquisition unit 12 acquire input time-series data A and output finger information B of training data D, respectively.

[0068] Fig. 10 is a diagram showing an example of input time series data A in the second embodiment. The upper part of Fig. 10 shows input time series data Az described in a format for expressing musical scores. The lower part of Fig. 10 shows the musical score represented by the input time series data Az.

[0069] As shown in the upper part of Fig. 10, the input time series data Az includes a plurality of elements A0 to A24. The elements A0 to A24 in Fig. 10 are similar to the elements A0 to A24 in the modified example (Fig. 8) in the first embodiment. The input time series data Az also includes additional elements arranged immediately after some of the elements A5, A9, A11, A13, A16, A18, and A20 corresponding to the notes. In the example of Fig. 10, the input time series data Az further includes elements A5f, A11f, A16f, and A20f arranged immediately after the elements A5, A11, A16, and A20, respectively.

[0070] The elements A5f, A11f, A16f, and A20f are finger information (hereinafter referred to as basic finger information) indicating the finger numbers to be used when playing the notes corresponding to the immediately preceding elements A5, A11, A16, and A20 on an instrument. The finger numbers are specified by the "finger" attribute in the elements A5f, A11f, A16f, and A20f. Therefore, the elements A5f, A11f, A16f, and A20f write the finger numbers "1," "2," "3," and "2" to be used when playing the notes N1, N3, N5, and N7 on the musical score, respectively, as shown in the lower part of FIG.

[0071] The output finger information Bz in this embodiment is the same as the output finger information Bz in the modified example (FIG. 9) in the first embodiment. Therefore, the first acquisition unit 11 can acquire the input time series data Az by randomly deleting a part of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz. The ratio of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f to be deleted can be specified by the user of the training device 10 by operating the operation unit 150 in FIG. 1.

[0072] In this example, the elements B9f, B13f, and B18f are deleted from the output finger information Bz to obtain the input time-series data Az. The elements B5f, B11f, B16f, and B20f that are not deleted remain as the elements A5f, A11f, A16f, and A20f of the basic finger information.

[0073] The construction unit 13 in FIG. 3 performs machine learning using the above input time-series data Az as an input element and output finger information Bz as an output element. By repeating machine learning for a plurality of training data D, a trained model M showing the input-output relationship between the input time-series data Az and the output finger information Bz is constructed.

[0074] In the finger movement presentation device 20, the reception unit 21 receives time-series data. The time-series data further includes basic finger information indicating the fingers used when playing some of the notes included in the musical note sequence with a musical instrument. The estimation unit 22 estimates finger information indicating the fingers used when playing the notes included in the musical note sequence with a musical instrument based on the constructed trained model M and the basic finger information. The generation unit 23 generates musical score information based on the musical note sequence of the time-series data and the finger information.

[0075] (2) Effects of the Embodiment According to the present embodiment, even when only the finger information (basic finger information) for some of the notes included in the musical note sequence of the time-series data is known and the finger information for the remaining notes is not given, the finger information for the remaining notes is complemented. Thereby, appropriate finger movements when a beginner-level player plays a musical instrument can be presented. The generation unit 23 may generate a video file showing the movement of the fingers by animation or the like based on the finger information estimated by the estimation unit 22. In this case, the movement of the fingers can be visualized.

[0076] (3) Modification Example In the present embodiment, the estimation unit 22 estimates the finger information for all the notes included in the musical note sequence of the time-series data, but the embodiment is not limited to this. When finger information is given for the notes of the first ratio among the notes included in the musical note sequence, the estimation unit 22 may estimate the finger information for the notes of the second ratio that is larger than the first ratio among the notes included in the musical note sequence. In this case, appropriate finger movements when a beginner-level or intermediate-level player plays a musical instrument can be presented.

[0077] In a modified example, the output finger information B of the training data D may not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. For example, when the input time series data Az includes the elements A5f, A11f, A16f, and A20f, the output finger information B includes the elements B5f, B11f, B16f, and B20f. On the other hand, the output finger information B may not include some of the elements B9f, B13f, and B18f.

[0078] [3] Third embodiment (1) Processing system The processing system 100 in the third embodiment will be described with respect to differences from the processing system 100 in the first embodiment. In this embodiment, the training data D indicates a set of input time series data A and output note information. In the training device 10 in Fig. 3, the first acquisition unit 11 and the second acquisition unit 12 acquire the input time series data A and the output note information of the training data D, respectively. The acquisition of the output note information is executed in place of step S2 in the sound learning process in Fig. 5.

[0079] The input time series data Az in this embodiment is the same as the input time series data Az in the modified example (FIG. 8) in the first embodiment. The first acquisition unit 11 can acquire the input time series data Az by deleting the elements C9f, C11f, and C16f from the output note information Cz in FIG. 11, which will be described later.

[0080] Fig. 11 is a diagram showing an example of output note information C in the third embodiment. The upper part of Fig. 11 shows output note information C(Cz) described in a format for expressing musical scores. The lower part of Fig. 11 shows the musical score represented by the output note information Cz.

[0081] As shown in the upper part of Fig. 11, the output note information Cz includes a plurality of elements C0 to C24. The elements C0 to C24 in Fig. 11 are the same as the elements B0 to B24 of the output fingering information Bz in the modification (Fig. 9) in the first embodiment. The output note information Cz also includes additional elements arranged immediately after some of the elements C5, C9, C11, C13, C16, C18, and C20 corresponding to the notes.

[0082] In this example, the attribute of "fingerrate" in the element C0 is "2", and the attribute "2" means a ratio of 40%. Therefore, the output note information Cz further includes elements C9f, C11f, and C16f, which are respectively arranged immediately after the elements C9, C11, and C16, which are approximately 40% of the elements among the elements C5, C9, C11, C13, C16, C18, and C20.

[0083] The elements C9f, C11f, and C16f indicate the notes corresponding to the immediately preceding elements C9, C11, and C16, respectively, as the notes to which fingering is to be assigned from the reference note sequence. As shown in the lower part of Fig. 11, the elements C9f, C11f, and C16f allow the notes N2, N3, and N5 corresponding to the elements C9, C11, and C16 to be written in the musical score in an identifiable manner.

[0084] 3 performs machine learning using the above input time series data Az as an input element and the output note information Cz as an output element. By repeating machine learning for multiple training data D, a trained model M showing the input / output relationship between the input time series data Az and the output note information Cz is constructed.

[0085] In the fingering presentation device 20, the receiving unit 21 receives time-series data. The estimating unit 22 estimates note information indicating notes to which fingering is to be assigned from a sequence of notes, based on the trained model M constructed by the training device 10 and the time-series data received by the receiving unit 21. The estimation of the note information is executed in place of step S12 in the fingering presentation process of Fig. 6. The generating unit 23 generates musical score information indicating an auxiliary musical score in which the notes indicated by the note information are displayed in an identifiable manner.

[0086] (2) Effects of the embodiment According to this embodiment, it is possible to present the notes to which fingering should be assigned from the sequence of notes, thereby enabling beginner or intermediate level players to recognize the key notes when playing an instrument.

[0087] (3) Variations The estimation unit 22 may estimate fingering information indicating fingers to be used when playing some notes included in the sequence of notes on an instrument, using the first trained model M constructed in the first embodiment and the second trained model M constructed in the present embodiment. Fig. 12 is a flowchart showing an example of a fingering suggestion process in the modified example.

[0088] First, the receiving unit 21 receives time-series data (step S21). Next, the estimating unit 22 estimates middle fingering information from the time-series data received in step S11 using the first trained model M constructed in the first embodiment (step S22). The middle fingering information indicates the fingers to be used when playing each note included in the sequence of notes with a musical instrument.

[0089] In addition, the estimation unit 22 performs step S using the second trained model M constructed in this embodiment. 21 The note information is estimated from the received time-series data (step S23). Steps S22 and S23 may be executed either first or simultaneously.

[0090] Next, the estimation unit 22 estimates fingering information for notes included in the sequence of notes other than the notes indicated by the note information estimated in step S23 based on the middle fingering information estimated in step S22 (step S24). After that, the generation unit 23 generates musical score information based on the sequence of notes in the time-series data accepted in step S21 and the fingering information estimated in step S24 (step S25). This ends the fingering suggestion process.

[0091] In this fingering guidance presentation process, the middle fingering information estimated in step S22 has a similar configuration to the output fingering information Bz in the modified example of the first embodiment (FIG. 9), for example. Also, the note information estimated in step S23 has a similar configuration to the output note information Cz in FIG. 11. FIG. 13 is a diagram showing an example of fingering information estimated in step S24 of the fingering guidance presentation process.

[0092] The upper part of Fig. 13 shows fingering information F(Fz) described by a method for expressing musical scores. The lower part of Fig. 13 shows an auxiliary musical score represented by fingering information Fz. The fingering information Fz is estimated by deleting elements B9f, B11f, and B16f, which respectively correspond to elements C9f, C11f, and C16f, which indicate the notes to which fingering is to be assigned in the note information (see Fig. 11), from the intermediate fingering information (see Fig. 9).

[0093] Specifically, as shown in the upper part of Fig. 13, the fingering information Fz includes a plurality of elements F1 to F24. The elements F1 to F24 in Fig. 13 are respectively similar to the elements B1 to B24 of the output fingering information Bz in the modified example (Fig. 9) in the first embodiment. The fingering information Fz also includes additional elements arranged immediately after some of the elements F5, F9, F11, F13, F16, F18, and F20 corresponding to the notes. In this example, the fingering information Fz further includes elements F5f, F13f, F18f, and F20f arranged immediately after the elements F5, F13, F18, and F20, respectively.

[0094] The elements F5f, F13f, F18f, and F20f indicate the finger numbers to be used when playing the notes corresponding to the immediately preceding elements F5, F13, F18, and F20, respectively, on an instrument. The finger numbers are specified by the "finger" attribute in the elements F5f, F13f, F18f, and F20f. Thus, the elements F5f, F13f, F18f, and F20f write the finger numbers "1", "1", "3", and "2" to be used when playing the notes N1, N4, N6, and N7, respectively, in the auxiliary score, as shown in the lower part of Figure 13.

[0095] According to the modified example, fingering information for some notes is thinned out from fingering information for all notes included in the note sequence of the time-series data. In this case, it is possible to present appropriate fingering when a beginner or intermediate level player plays an instrument. For example, fingering information for key notes when playing an instrument is thinned out, so that a beginner or intermediate level player can develop the ability to judge appropriate fingering when practicing an instrument.

[0096] [4] Other embodiments In the above embodiment, the fingering presentation device 20 includes the generation unit 23, but the embodiment is not limited to this. The performer can create an auxiliary score by transcribing the fingering information estimated by the estimation unit 22 to a desired score. Therefore, the fingering presentation device 20 does not need to include the generation unit 23.

[0097] In the above embodiment, the training data D is trained to estimate finger information when playing a piano, but the embodiment is not limited to this. The training data D may be trained to estimate finger information when playing other instruments such as drums.

[0098] In the above embodiment, the user of the fingering presentation device 20 is a performer, but the user of the fingering presentation device 20 may be, for example, a staff member of a music score production company. Furthermore, the machine learning by the training device 10 may be performed in advance by a staff member of the music score production company.

Claims

1. A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes; an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model, The time-series data further includes a performer identifier indicating a performer who performs the sequence of notes, The estimation unit estimates the fingering information based on the player identifier, the trained model is a machine learning model that has learned an input / output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument; the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes; The output fingering information indicates the fingers of the reference player to be used when playing a note on the musical instrument.

2. The estimation unit further estimates note information indicating a note to which fingering is to be assigned from the sequence of notes using the trained model; The fingering guidance presentation device according to claim 1, wherein the trained model is a machine learning model that further acquires an input / output relationship between the input time series data and output note information indicating notes to which fingerings are to be assigned from the reference note sequence.

3. 3. The fingering guidance suggestion device according to claim 1, wherein the player identifier is determined so as to correspond to a physical characteristic of the player.

4. 4. The fingering suggestion device according to claim 1, wherein the performer identifier is determined so as to correspond to a style of performance by the performer.

5. A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes; an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model, the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of notes included in the sequence of notes on a musical instrument, the estimation unit estimates, based on the basic fingering information, the fingering information indicating a finger to be used when playing notes having a second proportion, which is greater than the first proportion, among the notes included in the sequence of notes, on a musical instrument; the trained model is a machine learning model that has learned an input / output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument; the input time-series data further includes the basic fingering information indicating fingers to be used when playing a first proportion of notes included in the reference sequence of notes on the musical instrument, The output fingering information further includes the basic fingering information included in the input time-series data.

6. A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes; an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model, and note information indicating notes to which fingerings are to be assigned from the sequence of notes, The fingering presentation device includes an estimation unit that estimates intermediate fingering information indicating the fingers to be used when playing each note included in the sequence of notes on an instrument, and the note information, and estimates the fingering information indicating the fingers to be used when playing notes included in the sequence of notes on an instrument other than the notes indicated by the note information, by deleting from the intermediate fingering information finger information corresponding to the notes to which fingering in the note information is to be assigned.

7. The fingering presentation device according to any one of claims 1 to 6, further comprising a generation unit that generates musical score information indicating a first auxiliary musical score to which the fingering information is assigned so as to correspond to at least some of the notes included in the sequence of notes.

8. A fingering presentation device as described in claim 2 or 6, further comprising a generation unit that generates music score information indicating a second auxiliary music score in which the notes indicated by the note information are displayed in an identifiable manner.

9. A first acquisition unit that acquires input time-series data including a reference note sequence consisting of a plurality of notes; a second acquisition unit that acquires output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument, or output note information indicating notes to which fingerings are to be assigned from the reference sequence of notes; a construction unit that constructs a trained model that has learned an input / output relationship between the input time-series data and the output fingering information or the output note information, the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes; The output fingering information indicates the reference player's fingers to use when playing on the musical instrument.

10. Accepts time series data including a sequence of multiple notes, using the trained model to estimate fingering information indicative of fingers to be used when playing at least some of the notes in the sequence of notes on a musical instrument; The time-series data further includes a performer identifier indicating a performer who performs the sequence of notes, estimating the finger information includes estimating the finger information based on the player identifier, the trained model is a machine learning model that has learned an input / output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument; the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes; the output fingering information indicates the fingers of the reference player to be used when playing a note on the musical instrument; A computer-implemented method for providing fingering instructions.

11. Accepts time series data including a sequence of multiple notes, using the trained model to estimate fingering information indicative of fingers to be used when playing at least some of the notes in the sequence of notes on a musical instrument; the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of notes included in the sequence of notes on a musical instrument, estimating the fingering information includes estimating, based on the basic fingering information, fingering information indicating fingers to be used when playing notes having a second proportion, which is greater than the first proportion, among the notes included in the sequence of notes, on a musical instrument; and the trained model is a machine learning model that has learned an input / output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument; the input time-series data further includes the basic fingering information indicating fingers to be used when playing a first proportion of notes included in the reference sequence of notes on the musical instrument, The output move information further includes the basic move information included in the input time-series data, A computer-implemented method for providing fingering instructions.

12. Accepts time series data including a sequence of multiple notes, using the trained model, estimating fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument, and note information indicating notes to which fingerings are to be assigned from the sequence of notes; estimating the fingering information includes estimating intermediate fingering information indicating fingers to be used when playing each note included in the sequence of notes on a musical instrument, and the note information, and deleting fingering information corresponding to a note to which fingering in the note information is to be assigned from the intermediate fingering information, thereby estimating the fingering information indicating fingers to be used when playing notes included in the sequence of notes, other than the notes indicated by the note information, on a musical instrument, A computer-implemented method for providing fingering instructions.

13. Obtaining input time series data including a reference sequence of a plurality of notes; obtain output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument, or output note information indicating notes to which fingering is to be assigned from the reference sequence of notes; Constructing a trained model that has learned the input / output relationship between the input time-series data and the output fingering information or the output note information; the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes; the output fingering information indicates fingers of the reference player to be used when playing the musical instrument, A computer-implemented training method.

Images