Fingering presentation device, training device, fingering presentation method and training method

Abstract

To provide a fingering presentation device, training device, fingering presentation method, and training method for presenting fingering positions when playing a musical instrument.SOLUTION: In a presented fingering presentation device 100 comprising a training device 10 and a fingering indication device 20, the fingering indication device includes a reception unit and an estimation unit. The reception unit receives time-series data including a sequence of multiple musical notes and a performer identifier indicating a performer who plays the sequence of musical notes. The estimation unit estimates musical note information using a trained model. The note information indicates the notes that are objects of fingering to be assigned from the note sequence based on the performer identifier. The trained model is a machine learning model that has learned an input-output relationship between input time-series data including a reference note sequence consisting of multiple notes and a reference performer identifier indicating a reference performer who plays the reference note sequence, and output note information indicating the notes that are objects of the fingering by the reference performer to be assigned from the reference note sequence.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a fingering presentation device, a training device, a fingering presentation method, and a training method for presenting fingering when playing a musical instrument.

Background Art

[0002] Devices for assisting in the practice of playing musical instruments are known. For example, in the information processing device described in Patent Document 1, the playing technique level of a player is calculated, and based on the calculated playing technique level, musical pieces that the player can play are presented. However, when the player is inexperienced, it is not easy to appropriately determine the fingerings (hereinafter referred to as fingering) when playing each note on the musical instrument. On the other hand, Patent Document 2 describes a fingering determination method for determining the fingering at each note of a musical note series based on a probability model.

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0003] According to Patent Document 2, a player can recognize the fingering in musical instrument performance based on a probability model. However, in reality, there are innumerable combinations of fingerings, and there is not just one optimal fingering for playing a musical piece. Therefore, it is desired that more appropriate fingering be presented.

[0004] An object of the present invention is to provide a fingering presentation device, a training device, a fingering presentation method, and a training method capable of presenting appropriate fingering when playing a musical instrument.

Means for Solving the Problems

[0005] The fingering suggestion device according to the first aspect of the present invention includes a receiving unit that receives time-series data including a musical note sequence composed of a plurality of musical notes and a performer identifier indicating a performer who performs the musical note sequence, and an estimating unit that estimates note information indicating a musical note to which fingering is to be assigned from the musical note sequence based on the performer identifier using a trained model. The trained model is a machine learning model that has learned the input-output relationship between input time-series data including a reference musical note sequence composed of a plurality of musical notes and a reference performer identifier indicating a reference performer who performs the reference musical note sequence, and output note information indicating a musical note to which the fingering of the reference performer is to be assigned from the reference musical note sequence.

[0006] The training device according to the second aspect of the present invention includes a first acquisition unit that acquires input time-series data including a reference musical note sequence composed of a plurality of musical notes and a reference performer identifier indicating a reference performer who performs the reference musical note sequence, a second acquisition unit that acquires output note information indicating a musical note to which the fingering of the reference performer is to be assigned from the reference musical note sequence, and a construction unit that constructs a trained model that has learned the input-output relationship between the input time-series data and the output note information.

[0007] The fingering suggestion method according to the third aspect of the present invention receives time-series data including a musical note sequence composed of a plurality of musical notes and a performer identifier indicating a performer who performs the musical note sequence, estimates note information indicating a musical note to which fingering is to be assigned from the musical note sequence based on the performer identifier using a trained model, the trained model is a machine learning model that has learned the input-output relationship between input time-series data including a reference musical note sequence composed of a plurality of musical notes and a reference performer identifier indicating a reference performer who performs the reference musical note sequence, and output note information indicating a musical note to which the fingering of the reference performer is to be assigned from the reference musical note sequence, and is executed by a computer.

[0008] The training method according to the fourth aspect of the present invention acquires input time-series data including a reference musical note sequence composed of a plurality of musical notes and a reference performer identifier indicating a reference performer who performs the reference musical note sequence, acquires output note information indicating a musical note to which the fingering of the reference performer is to be assigned from the reference musical note sequence, constructs a trained model that has learned the input-output relationship between the input time-series data and the output note information, and is executed by a computer. [Effect of the Invention]

[0009] According to the present invention, appropriate fingerings when playing a musical instrument can be presented. [Brief Description of the Drawings]

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

[0011] [1] First Embodiment (1) Configuration of the Processing System Hereinafter, an operation instruction presentation device, a training device, an operation instruction presentation method, and a training method according to an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of a processing system including an operation instruction presentation device and a training device according to the first embodiment of the present invention. As shown in FIG. 1, the processing system 100 includes a RAM (Random Access Memory) 110, a ROM (Read Only Memory) 120, a CPU (Central Processing Unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.

[0012] The processing system 100 is realized by a computer such as a personal computer, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of a plurality of computers connected by a communication path such as Ethernet, or may be realized by an electronic musical instrument having a performance function such as an electronic piano.

[0013] The RAM 110, ROM 120, CPU 130, storage unit 140, operation unit 150, and display unit 160 are connected to a bus 170. The training device 10 and the operation instruction presentation device 20 are constituted by the RAM 110, ROM 120, and CPU 130. In the present embodiment, the training device 10 and the operation instruction presentation device 20 are constituted by a common processing system 100, but may be constituted by separate processing systems.

[0014] The RAM 110 is composed of, for example, a volatile memory and is used as a work area for the CPU 130. The ROM 120 is composed of, for example, a non-volatile memory and stores a training program and an operation instruction presentation program. The CPU 130 performs training processing by executing the training program stored in the ROM 120 on the RAM 110. Further, the CPU 130 performs operation instruction presentation processing by executing the operation instruction presentation program stored in the ROM 120 on the RAM 110. Details of the training processing and the operation instruction presentation processing will be described later.

[0015] The training program or fingering guidance program may be stored not in the ROM 120 but in the storage unit 140. Alternatively, the training program or fingering guidance program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the training program or fingering guidance program distributed from a server (including a cloud server) on the network may be installed in the ROM 120 or the storage unit 140.

[0016] The storage unit 140 includes storage media such as a hard disk, an optical disk, a magnetic disk, or a memory card, and stores the trained model M and a plurality of training data D. The trained model M or each training data D may not be stored in the storage unit 140 but may be stored in a computer-readable storage medium. Alternatively, when the processing system 100 is connected to a network, the trained model M or each training data D may be stored in a server on the network.

[0017] (2) Training Data The trained model M is a machine learning model trained to present fingering when a user of the fingering guidance device 20 (hereinafter referred to as a performer) plays a piece of music with an instrument, and is constructed using a plurality of training data D. The user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is data created based on the performance knowledge or performance style of a reference performer, etc. The reference performer has relatively high skills in playing a piece of music. The reference performer may be an instructor or a teacher of a performer in playing a piece of music.

[0018] The training data D represents a pair of input time-series data and output finger information. The input time-series data represents a reference note sequence consisting of a plurality of musical notes. The input time-series data may be image data representing an image of a musical score. The output finger information indicates the fingers of a reference performer used when playing each note of the reference note sequence and can be used to present the fingering when playing the reference note sequence. The output finger information may be a unique number assigned to each finger. In this example, numbers "1" to "5" are respectively assigned to the thumb, index finger, middle finger, ring finger, and little finger.

[0019] Here, the optimal fingering for playing a piece of music varies depending on the physical characteristics of the performer or the performance style of the performer. Therefore, in the present embodiment, the input time-series data further includes a reference performer identifier indicating the classification (category) of the reference performer who plays the reference note sequence. The reference performer identifier is determined to be different for at least one of the physical characteristics of the reference performer and the performance style of the reference performer. The physical characteristics of the reference performer include, for example, the size of the reference performer's hand (length of the fingers), age, gender, or the distinction between an adult and a child.

[0020] Figure 2 is a diagram showing an example of each training data D. The example in Figure 2 shows a part of the input time-series data and output finger information when a reference performer plays the piano. As shown in Figure 2, the input time-series data A includes elements A0 to A16. Element A0 corresponds to the reference performer identifier and is represented by a character string that is different for at least one of the physical characteristics of the reference performer and the performance style of the reference performer. Elements A1 to A16 correspond to the reference note sequence. In this example, element A0 is placed at the beginning of the input time-series data A, that is, before the reference note sequence (elements A1 to A16), but it may be placed at any position in the input time-series data A.

[0021] In elements A1, A3, A5, …, A15, “L” means the left hand, the number means the number assigned to the key, and “on” and “off” mean key press and key release respectively. In elements A2, A4, A6, …, A16, “wait” means standby, and the number means the length of time. Therefore, elements A1 to A4 mean pressing the key numbered “66” and holding it for 13 unit times, then releasing the key numbered “66” and holding it for 2 unit times.

[0022] The output finger information B includes elements B0 to B16 corresponding to elements A0 to A16 of the input time-series data A respectively. Element B0 indicates the reference performer identifier and is represented by the same character string as element A0. In elements B1, B3, B5, …, B15, “L” means the left hand, the number means the number assigned to the finger, and “down” and “up” mean push up and push down respectively. In elements B2, B4, B6, …, B16, “wait” means standby, and the number means the length of time. Therefore, elements B1 to B4 mean pushing down the middle finger of the left hand and waiting for 13 unit times, then pushing up the middle finger of the left hand and holding it for 2 unit times.

[0023] The training data D in FIG. 2 is generated to show the finger movement of the left hand, but the embodiment is not limited thereto. The training data D may be generated to show the finger movement of the right hand, or may be generated to show the finger movement of each of the left hand and the right hand. In the elements of the input time-series data A and the output finger information B for showing the finger movement of the right hand, for example, “R” instead of the character “L” may be used.

[0024] (3) Training device and finger movement presentation device FIG. 3 is a block diagram showing the configuration of the training device 10 and the finger movement presentation device 20. As shown in FIG. 3, the training device 10 includes, as functional units, a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13. By the CPU 130 in FIG. 1 executing the training program, the functional units of the training device 10 are realized. At least a part of the functional units of the training device 10 may be realized by hardware such as an electronic circuit.

[0025] The first acquisition unit 11 acquires the input time-series data A from each training data D stored in the storage unit 140 or the like. The second acquisition unit 12 acquires the output target information B from each training data D. The construction unit 13 performs machine learning on each training data D, using the input time-series data A acquired by the first acquisition unit 11 as the input element and the output target information B acquired by the second acquisition unit 12 as the output element. By repeating machine learning for a plurality of training data D, the construction unit 13 constructs a trained model M indicating the input-output relationship between the input time-series data A and the output target information B.

[0026] In this example, the construction unit 13 constructs the trained model M by training a Transformer, but the embodiment is not limited to this. The construction unit 13 may construct the trained model M by training a machine learning model of another method for handling time series. The trained model M constructed by the construction unit 13 is stored, for example, in the storage unit 140. The trained model M constructed by the construction unit 13 may be stored in a server or the like on the network.

[0027] The fingering presentation device 20 includes, as functional units, a reception unit 21, an estimation unit 22, and a generation unit 23. When the CPU 130 in FIG. 1 executes the fingering presentation program, the functional units of the fingering presentation device 20 are realized. At least a part of the functional units of the fingering presentation device 20 may be realized by hardware such as an electronic circuit.

[0028] In this embodiment, the reception unit 21 receives time-series data including a note sequence composed of a plurality of notes. A performer can provide the reception unit 21 with image data indicating an image of a musical score as the time-series data. Alternatively, the performer can generate time-series data by operating the operation unit 150 and provide it to the reception unit 21.

[0029] In this example, the time series data has the same configuration as the input time series data A in FIG. 2, and further includes a performer identifier indicating the classification (category) of the performer who plays the musical note sequence. The performer identifier is determined to be different for at least one of the physical characteristics of the performer and the playing style by the performer. The physical characteristics of the performer include, for example, the size of the performer's hand, age, gender, or the distinction between an adult and a child.

[0030] The estimation unit 22 estimates finger information using the trained model M stored in the storage unit 140 or the like. The finger information indicates the fingers of the performer used when playing each musical note of the musical note sequence received by the reception unit 21, and is estimated based on the musical note sequence and the performer identifier. The finger information may be a unique number assigned to each finger. The generation unit 23 generates score information based on the musical note sequence of the time series data received by the reception unit 21 and the finger information estimated by the estimation unit 22.

[0031] An auxiliary score is displayed on the display unit 160 based on the score information generated by the generation unit 23. FIG. 4 shows an example of the auxiliary score displayed on the display unit 160. As shown in FIG. 4, on the auxiliary score, the finger information estimated by the estimation unit 22 is shown so as to correspond to each musical note of the musical note sequence received by the reception unit 21. In the example of FIG. 4, the numbers of the fingers of one hand are shown as the finger information.

[0032] When distinguishing between the numbers of the fingers of the left hand or the right hand, a predetermined character such as "L" may be attached near the number of the finger of the left hand, and another predetermined character such as "R" may be attached near the number of the finger of the right hand. Alternatively, a predetermined color such as red may be attached to the number of the finger of the left hand or the musical note corresponding thereto, and another predetermined color such as blue may be attached to the number of the finger of the right hand or the musical note corresponding thereto.

[0033] (4) Training process and fingering presentation process FIG. 5 is a flowchart showing an example of the training process by the training device 10 in FIG. 3. The training process in FIG. 5 is performed by the CPU 130 in FIG. 1 executing a training program. First, the first acquisition unit 11 acquires the input time-series data A from each training data D (step S1). Also, the second acquisition unit 12 acquires the output target information B from each training data D (step S2). Either of steps S1 and S2 may be executed first, or they may be executed simultaneously.

[0034] Next, the construction unit 13 performs machine learning on each training data D, using the input time-series data A acquired in step S1 as the input element and the output target information B acquired in step S2 as the output element (step S3). Subsequently, the construction unit 13 determines whether sufficient machine learning has been executed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeated while the parameters are changed until sufficient machine learning is executed. The number of repetitions of the machine learning changes according to the quality conditions that the trained model M to be constructed should satisfy.

[0035] When sufficient machine learning has been executed, the construction unit 13 saves the input-output relationship between the input time-series data A and the output target information B learned by the machine learning in step S3 as the trained model M (step S5). Thereby, the training process ends.

[0036] FIG. 6 is a flowchart showing an example of the operation target presentation process by the operation target presentation device 20 in FIG. 3. The operation target presentation process in FIG. 6 is performed by the CPU 130 in FIG. 1 executing an operation target presentation program. First, the reception unit 21 receives time-series data (step S11). Next, the estimation unit 22 estimates the target information from the time-series data received in step S11, using the trained model M saved in step S5 of the training process (step S12).

[0037] After that, the generation unit 23 generates score information based on the note sequence of the time series data received in step S11 and the finger information estimated in step S12 (step S13). Based on the generated score information, an auxiliary score may be displayed on the display unit 160. Thereby, the finger movement presentation process ends.

[0038] (5) Effects of the Embodiment As described above, the finger movement presentation device 20 according to the present embodiment includes a reception unit 21 that receives time series data including a note sequence composed of a plurality of notes, and an estimation unit 22 that estimates finger information indicating a finger used when each note of the note sequence is played by an instrument using the trained model M. According to this configuration, appropriate finger information is estimated from the temporal flow of a plurality of notes in the time series data using the trained model M. Thereby, it is possible to present appropriate finger movement when playing a musical instrument.

[0039] The trained model M may be a machine learning model that has learned the input-output relationship between the input time series data A including a reference note sequence composed of a plurality of notes and the output finger information B indicating a finger used when each note of the reference note sequence is played by an instrument. In this case, finger information can be easily estimated from the time series data.

[0040] The time series data may further include a player identifier indicating a player who plays the note sequence, and the estimation unit 22 may estimate finger information based on the player identifier. In this case, appropriate finger information can be estimated according to the player.

[0041] The player identifier may be determined to correspond to the physical characteristics of the player. In this case, appropriate finger information can be estimated according to the physical characteristics of the player.

[0042] The player identifier may be determined to correspond to the playing style of the player. In this case, appropriate finger information can be estimated according to the playing style of the player.

[0043] The fingering prompt device 20 may further include a generation unit 23 that generates score information indicating an auxiliary score with finger information attached thereto corresponding to each note of the note sequence. In this case, the performer can easily recognize the finger corresponding to each note of the note sequence by visually recognizing the auxiliary score.

[0044] The training device 10 according to the present embodiment includes a first acquisition unit 11 that acquires input time-series data A including a reference note sequence composed of a plurality of notes, a second acquisition unit 12 that acquires output finger information B indicating fingers used when each note of the reference note sequence is played by an instrument, and a construction unit 13 that constructs a trained model M that has learned the input-output relationship between the input time-series data A and the output finger information B. According to this configuration, the trained model M that has learned the input-output relationship between the input time-series data A and the output finger information B can be easily constructed.

[0045] (6) Another example of training data In the present embodiment, the input time-series data A includes a reference performer identifier, and the time-series data includes a performer identifier, but the embodiment is not limited thereto. The input time-series data A only needs to include a reference note sequence and may not include a reference performer identifier. Similarly, the time-series data only needs to include a note sequence and may not include a performer identifier.

[0046] Also, in the present embodiment, the input time-series data A and the output finger information B are described in a so-called operation-based manner indicating key presses or key releases in the MIDI (Musical Instrument Digital Interface) standard, but the embodiment is not limited thereto. The input time-series data A and the output finger information B may be described in other formats. For example, the input time-series data A and the output finger information B may be described in a so-called note-based manner indicating the start position of a note or the length of a note in the MIDI standard. The same applies to the time-series data and the finger information.

[0047] FIG. 7 is a diagram showing another example of the input time-series data A. In the upper part of FIG. 7, the input time-series data A (Ax) described based on the operation is shown. In the middle part of FIG. 7, the input time-series data A (Ay) described based on the musical notes is shown. The input time-series data Ax and the input time-series data Ay include the same reference musical note sequence (the reference musical note sequence in the musical score shown in the lower part of FIG. 7). "Bar" and "beat" in the input time-series data Ax and Ay are elements indicating the beat structure of the reference musical note sequence.

[0048] As shown in FIG. 7, by describing the input time-series data A based on the musical notes, the length of the input time-series data A is shortened. Thereby, it becomes possible to easily process a longer input time-series data A. Note that the output finger information B corresponding to the input time-series data A can be described by inserting an element indicating the finger number immediately after an element ("note_○○") indicating the pitch number in the input time-series data A.

[0049] Alternatively, the input time-series data A and the output finger information B may be described by a method representing the musical score. Details of the input time-series data A and the output finger information B described by the method representing the musical score will be described in the following modified example.

[0050] (7) Modified Example FIG. 8 is a diagram showing an example of the input time-series data A in the modified example. In the upper part of FIG. 8, the input time-series data A (Az) described by the method representing the musical score is shown. In the lower part of FIG. 8, the musical score represented by the input time-series data Az is shown. As shown in the upper part of FIG. 8, the input time-series data Az includes a plurality of elements A0 to A24. Some elements have attributes. The attributes of the elements are described at the rear part (after the underscore) of the element.

[0051] Element A0 indicates the ratio of the notes to which fingering is assigned among the notes included in the reference note sequence. Although element A0 is placed at the beginning in the input time-series data Az, it may be placed at any position in the input time-series data Az. The ratio is specified by the "fingerrate" attribute in element A0. The attribute "5" in this example means a ratio of 100%. The ratio may have a range, for example, 20 - 40% or 40 - 60%, or may be divided into multiple ranges.

[0052] Element A1 indicates the part. Although element A1 is placed immediately after element A0, it may be placed at any position in the input time-series data Az. As element A1, "R" and "L" indicate the right - hand part and the left - hand part, respectively. In this example, elements corresponding to the right hand are placed after "R". Then "L" is placed, and elements corresponding to the left hand are placed after "L". The "R" and the elements corresponding to the right hand may be placed after the elements corresponding to the left hand. When there is no distinction between parts, the input time - series data Az does not include element A1.

[0053] Elements A2, A15, A24 indicate the bar lines of the musical score. Therefore, in the example of FIG. 8, the range delimited by "bar" in element A2 and "bar" in element A15 corresponds to the first measure. The range delimited by "bar" in element A15 and "bar" in element A24 corresponds to the second measure.

[0054] Element A3 indicates the clef of the musical score. The type of clef is specified by the "clef" attribute in element A3. In the example of FIG. 8, since the attribute is "treble", element A3 designates the treble clef as the clef. Note that when the attribute is "bass", element A3 designates the bass clef as the clef.

[0055] Element A4 indicates the time signature of the musical score. The type of time signature is specified by the "time" attribute in element A4. In the example of FIG. 8, since the attribute is "4 / 4", element A4 designates "4 / 4" as the time signature.

[0056] The notes in the reference note sequence are indicated by a pair of pitch and duration. The pitch is specified by the "note" attribute in elements A5, A9, A11, A13, A16, A18, A20. The duration is specified by the "len" attribute in elements A6, A10, A12, A14, A17, A19, A21. In this example, "len_1" corresponds to one beat.

[0057] The direction of the stem of a note in the musical score is specified by another attribute of "len" in elements A6, A10, A12, A14, A17, A19, A21. When the other attribute is "down", the stem extends downward from the note head. When the other attribute is "up", the stem extends upward from the note head. When a plurality of notes such as eighth notes or sixteenth notes are connected by a beam, the start position, relay position, and end position of the beam are specified by further other attributes "start", "continue", and "stop" of "len" in elements A10, A12, A14, respectively.

[0058] The rests in the reference note sequence are specified by "rest" in elements A7, A22. The duration of the rest is described by the "len" attribute in elements A8, A23.

[0059] In the example of FIG. 8, elements A5, A6 indicate note N1, and elements A7, A8 indicate rest R1. Elements A9, A10 indicate note N2, elements A11, A12 indicate note N3, and elements A13, A14 indicate note N4. Elements A16, A17 indicate note N5, elements A18, A19 indicate note N6. Elements A20, A21 indicate note N7, and elements A22, A23 indicate rest R2.

[0060] FIG. 9 is a diagram showing an example of the output finger information B in a modified example. In the upper part of FIG. 9, the output finger information B (Bz) described in a manner representing the musical score is shown. The output finger information Bz corresponds to the input time series data Az of FIG. 8. In the lower part of FIG. 9, the musical score represented by the output finger information Bz is shown.

[0061] As shown in the upper part of FIG. 9, the output finger information Bz includes a plurality of elements B0 to B24. Further, the output finger information Bz further includes elements B5f, B9f, B11f, B13f, B16f, B18f, B20f respectively arranged immediately after elements B5, B9, B11, B13, B16, B18, B20. Elements B0 to B24 are the same as elements A0 to A24 of the input time-series data Az in FIG. 8 respectively. Therefore, the first acquisition unit 11 in FIG. 3 can acquire the input time-series data Az by deleting elements B5f, B9f, B11f, B13f, B16f, B18f, B20f from the output finger information Bz.

[0062] Elements B5f, B9f, B11f, B13f, B16f, B18f, B20f respectively indicate the finger numbers used when playing the notes corresponding to the immediately preceding elements B5, B9, B11, B13, B16, B18, B20 with a musical instrument. The finger numbers are specified by the "finger" attribute in elements B5f, B9f, B11f, B13f, B16f, B18f, B20f. Therefore, as shown in the lower part of FIG. 9, the finger numbers "1", "1", "2", "1", "3", "3", and "2" used when playing notes N1 to N7 are respectively described in the musical score by elements B5f, B9f, B11f, B13f, B16f, B18f, B20f.

[0063] (VIII) Effects of the Modification In the modification of the first embodiment, the ratio of the notes to which fingering is assigned among the notes included in the reference note sequence can be arbitrarily specified according to the attribute of element A0. When the ratio is 100%, the estimation unit 22 estimates the finger information for all the notes included in the note sequence. In this case, appropriate fingering when a beginner-level player plays a musical instrument can be presented.

[0064] Also, when the ratio is 100%, the generation unit 23 may generate a video file that shows the movement of the finger by animation or the like based on the finger information estimated by the estimation unit 22. Thereby, the movement of the finger can be visualized. The generation of such a video file may be executed before or after step S13 in the finger movement presentation process of FIG. 6, may be executed in parallel with step S13, or may be executed in place of step S13.

[0065] On the other hand, when the ratio is less than 100%, the estimation unit 22 estimates some of the notes included in the note sequence that are the targets for assigning fingerings, and the finger information for the some notes. In this case, it is possible to present appropriate fingerings when a player at a beginner level or above, such as an elementary or intermediate level, plays a musical instrument. In this configuration, the output finger information Bz does not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, B20f.

[0066] Also, when the ratio is less than 100%, the estimation unit 22 may estimate note information indicating the notes that are the targets for assigning fingerings from the note sequence without estimating the finger information. Details will be described in the third embodiment described later.

[0067] [2] Second Embodiment (1) Processing System The differences between the processing system 100 in the second embodiment and the processing system 100 in the first embodiment will be described. In the training device 10 of FIG. 3, the first acquisition unit 11 and the second acquisition unit 12 respectively acquire the input time-series data A and the output finger information B of the training data D.

[0068] FIG. 10 is a diagram showing an example of the input time-series data A in the second embodiment. In the upper part of FIG. 10, the input time-series data Az described in a manner representing a musical score is shown. In the lower part of FIG. 10, the musical score represented by the input time-series data Az is shown.

[0069] As shown in the upper part of FIG. 10, the input time-series data Az includes a plurality of elements A0 to A24. The elements A0 to A24 in FIG. 10 are the same as the elements A0 to A24 in the modification example (FIG. 8) in the first embodiment, respectively. Further, the input time-series data Az includes additional elements arranged immediately after some of the elements A5, A9, A11, A13, A16, A18, A20 corresponding to musical notes. In the example of FIG. 10, the input time-series data Az further includes elements A5f, A11f, A16f, A20f arranged immediately after the elements A5, A11, A16, A20, respectively.

[0070] The elements A5f, A11f, A16f, A20f are finger information (hereinafter referred to as basic finger information) indicating the finger numbers used when playing the musical notes corresponding to the immediately preceding elements A5, A11, A16, A20 with a musical instrument. The finger numbers are specified by the "finger" attribute in the elements A5f, A11f, A16f, A20f. Therefore, as shown in the lower part of FIG. 10, the finger numbers "1", "2", "3", and "2" used when playing the musical notes N1, N3, N5, N7 are respectively described in the musical score by the elements A5f, A11f, A16f, A20f.

[0071] The output finger information Bz in the present embodiment is the same as the output finger information Bz in the modification example (FIG. 9) in the first embodiment. Therefore, the first acquisition unit 11 can acquire the input time-series data Az by randomly deleting some of the elements B5f, B9f, B11f, B13f, B16f, B18f, B20f from the output finger information Bz. The ratio of the elements B5f, B9f, B11f, B13f, B16f, B18f, B20f to be deleted can be specified by the user of the training device 10 operating the operation unit 150 in FIG. 1.

[0072] In this example, the input time-series data Az is acquired by deleting the elements B9f, B13f, B18f from the output finger information Bz. The elements B5f, B11f, B16f, B20f that are not deleted remain as the elements A5f, A11f, A16f, A20f which are basic finger information.

[0073] The construction unit 13 in FIG. 3 performs machine learning using the above input time-series data Az as an input element and output finger information Bz as an output element. By repeating machine learning for a plurality of training data D, a trained model M showing the input-output relationship between the input time-series data Az and the output finger information Bz is constructed.

[0074] In the fingering presentation device 20, the reception unit 21 receives time-series data. The time-series data further includes basic finger information indicating fingers used when playing some of the notes included in the note sequence with a musical instrument. The estimation unit 22 estimates finger information indicating fingers used when playing the notes included in the note sequence with a musical instrument based on the constructed trained model M and the basic finger information. The generation unit 23 generates score information based on the note sequence and finger information of the time-series data.

[0075] (2) Effects of the Embodiment According to the present embodiment, even when only finger information (basic finger information) for some of the notes included in the note sequence of the time-series data is known and finger information for the remaining notes is not given, the finger information for the remaining notes is complemented. Thereby, appropriate fingering when a beginner-level player plays a musical instrument can be presented. The generation unit 23 may generate a video file showing the movement of the fingers by animation or the like based on the finger information estimated by the estimation unit 22. In this case, the movement of the fingers can be visualized.

[0076] (3) Modifications In the present embodiment, the estimation unit 22 estimates finger information for all the notes included in the note sequence of the time-series data, but the embodiment is not limited to this. When finger information is given for a first ratio of the notes included in the note sequence, the estimation unit 22 may estimate finger information for a second ratio of the notes included in the note sequence that is greater than the first ratio. In this case, appropriate fingering when a beginner-level or intermediate-level player plays a musical instrument can be presented.

[0077] In the modification example, the output finger information B of the training data D may not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, B20f. For example, when the input time-series data Az includes the elements A5f, A11f, A16f, A20f, the output finger information B includes the elements B5f, B11f, B16f, B20f. On the other hand, the output finger information B may not include some of the elements B9f, B13f, B18f.

[0078] [3] Third Embodiment (1) Processing System Regarding the processing system 100 in the third embodiment, the differences from the processing system 100 in the first embodiment will be described. In this embodiment, the training data D represents a set of input time-series data A and output musical note information. In the training device 10 of FIG. 3, the first acquisition unit 11 and the second acquisition unit 12 respectively acquire the input time-series data A and the output musical note information of the training data D. The acquisition of the output musical note information is executed instead of step S2 in the acoustic learning process of FIG. 5.

[0079] The input time-series data Az in this embodiment is the same as the input time-series data Az in the modification example (FIG. 8) of the first embodiment. The first acquisition unit 11 can acquire the input time-series data Az by deleting the elements C9f, C11f, C16f from the output musical note information Cz of FIG. 11 described later.

[0080] FIG. 11 is a diagram showing an example of the output musical note information C in the third embodiment. In the upper part of FIG. 11, the output musical note information C (Cz) described in a notation representing a musical score is shown. In the lower part of FIG. 11, the musical score represented by the output musical note information Cz is shown.

[0081] As shown in the upper part of FIG. 11, the output note information Cz includes a plurality of elements C0 to C24. The elements C0 to C24 in FIG. 11 are the same as the elements B0 to B24 of the output finger information Bz in the modification (FIG. 9) in the first embodiment, respectively. Further, the output note information Cz includes additional elements arranged immediately after a part of the elements C5, C9, C11, C13, C16, C18, C20 corresponding to the notes.

[0082] In this example, the attribute of "fingerrate" in the element C0 is "2", and the attribute "2" means a ratio of 40%. Therefore, the output note information Cz further includes elements C9f, C11f, C16f arranged immediately after the elements C9, C11, C16, which are about 40% of the elements among C5, C9, C11, C13, C16, C18, C20.

[0083] The elements C9f, C11f, C16f respectively indicate the notes corresponding to the immediately preceding elements C9, C11, C16 as the notes to which fingerings are to be assigned from the reference note sequence. As shown in the lower part of FIG. 11, the notes N2, N3, N5 corresponding to the elements C9, C11, C16 are respectively described in the musical score in an identifiable manner by the elements C9f, C11f, C16f.

[0084] The construction unit 13 in FIG. 3 performs machine learning with the above input time-series data Az as the input element and the output note information Cz as the output element. By repeating machine learning for a plurality of training data D, a trained model M showing the input-output relationship between the input time-series data Az and the output note information Cz is constructed.

[0085] In the fingering presentation device 20, the reception unit 21 receives time-series data. The estimation unit 22 estimates note information indicating the notes to which fingerings are to be assigned from the note sequence based on the trained model M constructed by the training device 10 and the time-series data received by the reception unit 21. The estimation of the note information is executed instead of step S12 in the fingering presentation process of FIG. 6. The generation unit 23 generates score information indicating an auxiliary score in which the notes indicated by the note information are displayed in an identifiable manner.

[0086] (2) Effects of the Embodiment According to this embodiment, it is possible to present the notes to which fingering is to be assigned from the note sequence. As a result, beginner or intermediate-level players can recognize the crucial notes when playing the musical instrument.

[0087] (3) Modification The estimation unit 22 may estimate finger information indicating the fingers to be used when playing some of the notes included in the note sequence with a musical instrument, using the first trained model M constructed in the first embodiment and the second trained model M constructed in this embodiment. FIG. 12 is a flowchart showing an example of the fingering presentation process in the modification.

[0088] First, the reception unit 21 receives time-series data (step S21). Next, the estimation unit 22 estimates intermediate finger information from the time-series data received in step S11 using the first trained model M constructed in the first embodiment (step S22). The intermediate finger information indicates the fingers to be used when playing each note included in the note sequence with a musical instrument.

[0089] Also, the estimation unit 22 estimates note information from the time-series data received in step S21 using the second trained model M constructed in this embodiment (step S23). Steps S22 and S23 may be executed in either order or simultaneously.

[0090] Subsequently, the estimation unit 22 estimates finger information for the notes other than the notes indicated by the note information estimated in step S23 among the notes included in the note sequence, based on the intermediate finger information estimated in step S22 (step S24). Thereafter, the generation unit 23 generates score information based on the note sequence of the time-series data received in step S21 and the finger information estimated in step S24 (step S25). Thus, the fingering presentation process ends.

[0091] In this fingering guidance process, the intermediate finger information estimated in step S22 has the same configuration as, for example, the output finger information Bz in the modification (Fig. 9) of the first embodiment. Also, the note information estimated in step S23 has the same configuration as the output note information Cz in Fig. 11. Fig. 13 is a diagram showing an example of the finger information estimated in step S24 of the fingering guidance process.

[0092] In the upper part of Fig. 13, finger information F (Fz) described in a musical score representation method is shown. In the lower part of Fig. 13, an auxiliary musical score represented by the finger information Fz is shown. The finger information Fz is estimated by deleting elements B9f, B11f, and B16f corresponding to elements C9f, C11f, and C16f, which indicate the notes to which fingering in the note information (see Fig. 11) is to be applied, respectively, from the intermediate finger information (see Fig. 9).

[0093] Specifically, as shown in the upper part of Fig. 13, the finger information Fz includes a plurality of elements F1 to F24. The elements F1 to F24 in Fig. 13 are the same as the elements B1 to B24 of the output finger information Bz in the modification (Fig. 9) of the first embodiment, respectively. Also, the finger information Fz includes additional elements arranged immediately after some of the elements F5, F9, F11, F13, F16, F18, and F20 corresponding to notes. In this example, the finger information Fz further includes elements F5f, F13f, F18f, and F20f arranged immediately after the elements F5, F13, F18, and F20, respectively.

[0094] The elements F5f, F13f, F18f, and F20f each indicate the finger number used when playing the notes corresponding to the immediately preceding elements F5, F13, F18, and F20 on a musical instrument. The finger number is specified by the "finger" attribute in the elements F5f, F13f, F18f, and F20f. Therefore, by the elements F5f, F13f, F18f, and F20f, as shown in the lower part of Fig. 13, the finger numbers "1", "1", "3", and "2" used when playing the notes N1, N4, N6, and N7 are described in the auxiliary musical score, respectively.

[0095] According to the modified example, from the finger information for all the notes included in the note sequence of the time-series data, the finger information for some of the notes is thinned out. In this case, it is possible to present appropriate fingerings when a beginner or intermediate-level player plays a musical instrument. For example, since the finger information for the notes that are crucial when playing a musical instrument is thinned out, a beginner or intermediate-level player can cultivate the judgment ability of appropriate fingerings when practicing the musical instrument.

[0096] [4]Other embodiments In the above embodiment, the fingering presentation device 20 includes the generation unit 23, but the embodiment is not limited to this. A player can create an auxiliary musical score by transcribing the finger information estimated by the estimation unit 22 onto a desired musical score. Therefore, the fingering presentation device 20 may not include the generation unit 23.

[0097] In the above embodiment, the training data D is trained to estimate finger information when performing with a piano, but the embodiment is not limited to this. The training data D may be trained to estimate finger information when performing with other musical instruments such as drums.

[0098] In the above embodiment, the case where the user of the fingering presentation device 20 is a player has been described as an example, but the user of the fingering presentation device 20 may be, for example, a staff member of a music score creation company. Also, the machine learning by the training device 10 may be performed in advance by the staff member of the music score creation company.

Claims

1. A receiving unit that receives time series data including a note sequence composed of a plurality of notes and a performer identifier indicating a performer who performs the note sequence; An estimating unit that estimates note information indicating a note to which fingerings are to be assigned from the note sequence based on the performer identifier using a trained model; The trained model is a machine learning model that has learned the input-output relationship between input time series data including a reference note sequence composed of a plurality of notes and a reference performer identifier indicating a reference performer who performs the reference note sequence, and output note information indicating a note to which fingerings are to be assigned from the reference note sequence. A fingering presentation device.

2. The fingering presentation device according to claim 1, wherein the performer identifier is determined to correspond to a physical characteristic of the performer.

3. The fingering presentation device according to claim 1 or 2, wherein the performer identifier is determined to correspond to a performance style by the performer.

4. The fingering presentation device according to any one of claims 1 to 3, further comprising a generation unit that generates score information indicating an auxiliary score in which the notes indicated by the note information are visibly displayed.

5. A first acquisition unit that acquires input time series data including a reference note sequence composed of a plurality of notes and a reference performer identifier indicating a reference performer who performs the reference note sequence; A second acquisition unit that acquires output note information indicating a note to which fingerings are to be assigned from the reference note sequence by the reference performer; A training device comprising a construction unit that constructs a trained model that has learned the input-output relationship between the input time series data and the output note information.

6. Receiving time series data including a note sequence composed of a plurality of notes and a performer identifier indicating a performer who performs the note sequence, Estimating note information indicating a note to which fingerings are to be assigned from the note sequence based on the performer identifier using a trained model, The trained model is a machine learning model that has learned the input-output relationship between input time series data including a reference note sequence composed of a plurality of notes and a reference performer identifier indicating a reference performer who performs the reference note sequence, and output note information indicating a note to which fingerings are to be assigned from the reference note sequence, A fingering presentation method executed by a computer.

7. Acquiring input time series data including a reference note sequence composed of a plurality of notes and a reference performer identifier indicating a reference performer who performs the reference note sequence, Obtain output note information indicating notes to which the fingerings of the reference player are to be assigned from the reference note sequence, Construct a trained model that has learned the input-output relationship between the input time-series data and the output note information, A training method executed by a computer.

Images