Finger movement prompting device, training device, finger movement prompting method and training method

By constructing a trained model and using machine learning techniques, the problem of fingering prompts in musical instrument performance has been solved, achieving appropriate fingering prompts and improving performance efficiency.

CN116940978BActive Publication Date: 2026-11-10YAMAHA CORP
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202180095314.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-09
Filing Date
2021-11-01
Publication Date
2026-11-10
Estimated Expiration
2041-11-01

AI Technical Summary

Technical Problem

In musical instrument practice, current technology struggles to effectively guide performers on appropriate finger combinations in different situations.

Method used

By constructing a trained model, machine learning technology is used to infer the performer's finger information and provide prompts in combination with the musical score.

Benefits of technology

It enables appropriate fingering prompts during instrument playing, improving the performer's playing level and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116940978B_ABST
    Figure CN116940978B_ABST
Patent Text Reader

Abstract

The fingering prompting device has a receiving unit and an estimation unit. The receiving unit receives time-series data including a string of notes consisting of multiple notes. The estimation unit uses a trained model to estimate finger information. The finger information represents the finger used when playing at least a portion of the notes in the string of notes received by the receiving unit on an instrument. Alternatively, the estimation unit uses a trained model to estimate note information. The note information represents the note in the string of notes received by the receiving unit that becomes the object of the fingering.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a finger movement prompting device, training device, finger movement prompting method, and training method for prompting finger movements when playing a musical instrument. Background Technology

[0002] Devices for assisting in the practice of playing musical instruments are known. For example, in the information processing device described in Patent Document 1, the performer's playing skill level is calculated, and based on the calculated skill level, the performer is prompted with pieces they are capable of playing. However, when the performer is not proficient, it is not easy to properly determine the finger placement (hereinafter referred to as fingering) when playing each note on the instrument. In response, Patent Document 2 describes a fingering determination method that determines the fingering of each note in a note series based on a probability model.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-083845

[0004] Patent Document 2: Japanese Patent Application Publication No. 2007-241034 Summary of the Invention

[0005] According to Patent Document 2, the performer can identify fingering techniques in instrument playing based on a probability model. However, in reality, there are countless combinations of fingering techniques, and the optimal fingering technique for playing a piece of music is not singular. Therefore, it is desirable to suggest more appropriate fingering techniques.

[0006] The purpose of this invention is to provide a finger movement prompting device, a training device, a finger movement prompting method, and a training method that can prompt appropriate finger movements when playing a musical instrument.

[0007] According to one aspect of the present invention, a fingering prompting device comprises: a receiving unit that receives time-series data including a string of notes consisting of multiple notes; and an estimation unit that uses a trained model to estimate finger information or note information, wherein the finger information represents the finger used when playing at least a portion of the notes contained in the string of notes on an instrument, and the note information represents the note that becomes the object of fingering from the string of notes.

[0008] The training apparatus according to another aspect of the present invention comprises: a first acquisition unit that acquires input time-series data including a reference note string consisting of multiple notes; a second acquisition unit that acquires output finger information or output note information, the output finger information representing the finger used when playing at least a portion of the notes included in the reference note string by an instrument, the output note information representing the note to which the fingering is assigned from the reference note string; and a construction unit that constructs a trained model that has learned the input-output relationship between the input time-series data and the output finger information or output note information.

[0009] According to another embodiment of the invention, the fingering prompting method is executed by a computer, wherein time-series data including a string of notes consisting of multiple notes is received, and a trained model is used to estimate finger information or note information, the finger information representing the finger used when playing at least a portion of the notes contained in the string of notes on an instrument, and the note information representing the note that becomes the object of fingering from the string of notes.

[0010] According to another embodiment of the invention, the training method is performed by a computer, wherein input time-series data including a reference note string consisting of multiple notes is obtained, output finger information or output note information is obtained, the output finger information representing the finger used when playing at least a portion of the notes contained in the reference note string through an instrument, and the output note information representing the note that becomes the object of fingering from the reference note string, and a trained model is constructed that has learned the input-output relationship between the input time-series data and the output finger information or output note information.

[0011] The effects of the invention

[0012] According to the present invention, it is possible to provide guidance on appropriate fingering when playing a musical instrument. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating the structure of a processing system that includes the finger prompting device and training device according to the first embodiment of the present invention.

[0014] Figure 2 This is a graph representing an example of each training data point.

[0015] Figure 3 This is a block diagram showing the structure of the training device and the finger prompting device.

[0016] Figure 4 This represents an example of supplementary musical score displayed on the display section.

[0017] Figure 5 It means by Figure 3 A flowchart illustrating an example of training processing performed by a training device.

[0018] Figure 6 It means by Figure 3 A flowchart illustrating an example of finger prompting processing performed by a finger prompting device.

[0019] Figure 7 This is a graph representing other examples of input time series data.

[0020] Figure 8 This is a diagram representing an example of the input time series data for a variant example.

[0021] Figure 9 This is a diagram illustrating an example of the output finger information in a variant.

[0022] Figure 10 This is a diagram showing an example of the input time series data in the second embodiment.

[0023] Figure 11 This is a diagram illustrating an example of the output finger information in the third embodiment.

[0024] Figure 12 This is a flowchart illustrating an example of pointer hint processing for a variant.

[0025] Figure 13 This is a diagram illustrating an example of the finger information estimated in step S24 of the finger prompting process. Detailed Implementation

[0026] [1] First Embodiment

[0027] (1) Structure of the processing system

[0028] The following description, with reference to the accompanying drawings, details the finger-gesture prompting device, training device, finger-gesture prompting method, and training method involved in the embodiments of the present invention. Figure 1 This is a block diagram illustrating the structure of a processing system including the finger prompting device and training device according to the first embodiment of the present invention. For example... Figure 1 As shown, the processing system 100 includes RAM (Random Access Memory) 110, ROM (Read Only Memory) 120, and CPU (Central Processing Unit).

[0029] 130, storage unit 140, operation unit 150 and display unit 160.

[0030] The processing system 100 is implemented by a computer such as a personal computer, tablet terminal, or smartphone. Alternatively, the processing system 100 can be implemented by the joint operation of multiple computers connected by communication lines such as Ethernet, or by an electronic musical instrument with performance functions such as an electronic piano.

[0031] RAM 110, ROM 120, CPU 130, storage unit 140, operation unit 150, and display unit 160 are connected to bus 170. The training device 10 and the finger prompting device 20 are constituted by RAM 110, ROM 120, and CPU 130. In this embodiment, the training device 10 and the finger prompting device 20 are constituted by a common processing system 100, but they may also be constituted by different processing systems.

[0032] RAM 110, for example, is composed of volatile memory and serves as the operating area of ​​CPU 130. ROM 120, for example, is composed of non-volatile memory and stores the training program and the pointer prompt program. CPU 130 performs training processing by executing the training program stored in ROM 120 on RAM 110. Additionally, CPU 130 performs pointer prompt processing by executing the pointer prompt program stored in ROM 120 on RAM 110. Details of the training processing and pointer prompt processing will be described later.

[0033] The training program or command prompt program may not be stored in the ROM 120, but rather in the storage unit 140. Alternatively, the training program or command prompt program may be provided in a manner that allows it to be stored on a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the training program or command prompt program transmitted from a server (including a cloud server) on that network may be installed in the ROM 120 or the storage unit 140.

[0034] Storage unit 140 includes storage media such as hard disk, optical disk, magnetic disk, or memory card, and stores the trained model M and multiple training data D. The trained model M or each training data D may not be stored in storage unit 140, but rather in a computer-readable storage medium. Alternatively, when processing system 100 is connected to a network, the trained model M or each training data D may be stored on a server on that network.

[0035] (2) Training data

[0036] The trained model M is a machine learning model trained to provide finger movement prompts to the user of the finger movement prompting device 20 (hereinafter referred to as the performer) when playing a piece of music on an instrument. It is constructed using multiple training data sets D. The user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is created based on the performer's performance knowledge or performance style. The reference performer has a high level of skill in playing the piece of music. The reference performer can be an instructor or mentor to the performer of the piece of music.

[0037] Training data D represents the combination of input time-series data and output finger information. Input time-series data represents a reference note string consisting of multiple notes. Input time-series data can be image data representing a musical score. Output finger information represents the fingers of a reference performer used when playing each note of the reference note string on an instrument, serving as a cue for finger placement during performance. Output finger information can be assigned unique numbers to each finger. In this example, the thumb, index finger, middle finger, ring finger, and little finger are assigned numbers "1" through "5," respectively.

[0038] Here, the optimal fingering for playing the music varies depending on the performer's physical characteristics or playing style. Therefore, in this embodiment, the input time series data also includes a reference performer identifier representing the classification (category) of the reference performer playing the reference note string. The reference performer identifier is determined in a manner that varies based on at least one of the reference performer's physical characteristics and playing style. The physical characteristics of the reference performer include, for example, the size of the reference performer's hands (finger length), age, gender, or whether they are an adult or a child.

[0039] Figure 2 This is a graph representing an example of each training data D. Figure 2 The example represents a portion of the input time-series data and output finger information from a pianist playing the piano. For example... Figure 2 As shown, the input time series data A contains elements A0 to A16. Element A0 corresponds to the reference performer identifier and is represented by a string that differs based on at least one of the reference performer's physical characteristics and playing style. Elements A1 to A16 correspond to the reference note strings. In this example, element A0 is placed at the beginning of the input time series data A, i.e., before the reference note strings (elements A1 to A16), but it can also be placed at any position in the input time series data A.

[0040] In elements A1, A3, A5, ..., A15, "L" refers to the left hand, the number indicates the key number, and "on" and "off" refer to pressing and releasing the key, respectively. In elements A2, A4, A6, ..., A16, "wait" means to wait, and the number indicates the duration. Therefore, elements A1 to A4 refer to pressing and holding the key numbered "66" for 13 units of time, then releasing the key numbered "66" for 2 units of time.

[0041] The output finger information B contains elements B0 to B16 corresponding to elements A0 to A16 of the input time series data A, respectively. Element B0 represents the reference performer identifier, represented by the same string as element A0. In elements B1, B3, B5, ..., B15, "L" refers to the left hand, the numbers are the numbers assigned to the fingers, and "down" and "up" refer to lifting and pressing down, respectively. In elements B2, B4, B6, ..., B16, "wait" means waiting, and the numbers are the length of time. Therefore, elements B1 to B4 refer to pressing down the left middle finger, waiting for 13 units of time, then lifting the left middle finger and maintaining it for 2 units of time.

[0042] Figure 2 The training data D is generated to represent the finger movements of the left hand, but the implementation method is not limited to this. The training data D can be generated to represent the finger movements of the right hand, or it can be generated to represent the finger movements of both the left and right hands separately. In the elements of the input time series data A and the output finger information B used to represent the finger movements of the right hand, the character "L" may not be used, but for example, "R".

[0043] (3) Training device and finger prompting device

[0044] Figure 3 This is a block diagram showing the structure of the training device 10 and the finger prompting device 20. For example... Figure 3 As shown, the training device 10 includes a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13 as functional units. Figure 1 The CPU 130 executes the training program to implement the functional parts of the training device 10. At least a portion of the functional parts of the training device 10 can be implemented by hardware such as electronic circuits.

[0045] The first acquisition unit 11 acquires input time series data A from each training data D stored in the storage unit 140, etc. The second acquisition unit 12 acquires output finger information B from each training data D. The construction unit 13 performs machine learning on each training data D, setting the input time series data A acquired by the first acquisition unit 11 as input elements and the output finger information B acquired by the second acquisition unit 12 as output elements. By repeatedly performing machine learning on multiple training data D, the construction unit 13 constructs a trained model M representing the input-output relationship between the input time series data A and the output finger information B.

[0046] In this example, the construction unit 13 constructs a trained model M by training a Transformer, but the implementation is not limited to this. The construction unit 13 can construct a trained model M by training a machine learning model that processes time series data in other ways. The trained model M constructed by the construction unit 13 is stored, for example, in the storage unit 140. The trained model M constructed by the construction unit 13 can also be stored on a server on a network.

[0047] The finger prompting device 20 includes a receiving unit 21, an estimating unit 22, and a generating unit 23 as a functional part. Figure 1 The CPU 130 implements the functions of the pointer prompting device 20 by executing the pointer prompting program. At least a portion of the functions of the pointer prompting device 20 can be implemented by hardware such as electronic circuits.

[0048] In this embodiment, the receiving unit 21 receives time-series data including a string of notes consisting of multiple notes. The performer can supply image data representing a musical score to the receiving unit 21 as time-series data. Alternatively, the performer can generate time-series data by operating the operation unit 150 and supply it to the receiving unit 21.

[0049] In this example, time series data has the same characteristics as... Figure 2 The input time series data A has the same structure, but also includes performer identifiers representing the category (class) of the performers playing the note sequences. Performer identifiers are determined in a manner that varies based on the performer's physical characteristics and at least one of their playing styles. Physical characteristics include, for example, the size of the performer's hands, age, gender, or whether they are an adult or a child.

[0050] The estimation unit 22 uses a trained model M stored in the storage unit 140, etc., to estimate the finger information. The finger information represents the fingers of the performer used when playing each note of the note string received by the receiving unit 21, and is estimated based on the note string and the performer's identifier. The finger information can be a unique number assigned to each finger. The generation unit 23 generates musical score information based on the note string of time-series data received by the receiving unit 21 and the finger information estimated by the estimation unit 22.

[0051] The display unit 160 displays the auxiliary musical score based on the musical score information generated by the generation unit 23. Figure 4 This shows an example of supplementary musical score displayed on display unit 160. For example... Figure 4 As shown, in the auxiliary musical score, the finger information estimated by the estimation unit 22 is displayed in a manner corresponding to each note in the note string received by the receiving unit 21. Figure 4In the example, the finger numbers of a hand are shown as finger information.

[0052] When distinguishing between left-hand and right-hand finger numbers, a prescribed character such as "L" can be marked near the left-hand finger number, and a prescribed character such as "R" can be marked near the right-hand finger number. Alternatively, a prescribed color such as red can be used to mark the left-hand finger number or its corresponding note, and a prescribed color such as blue can be used to mark the right-hand finger number or its corresponding note.

[0053] (4) Training processing and operation prompt processing

[0054] Figure 5 It means Figure 3 A flowchart of an example of the training process of the training device 10. Figure 5 Training processing through Figure 1 The training program is executed by the CPU 130. First, the first acquisition unit 11 acquires the input time series data A from each training data D (step S1). Then, the second acquisition unit 12 acquires the output finger information B from each training data D (step S2). Regarding steps S1 and S2, either step S1 or step S2 can be executed first, or they can be executed simultaneously.

[0055] Next, the construction unit 13 performs machine learning on each training data D, setting the input time series data A obtained in step S1 as the input element and the output finger information B obtained in step S2 as the output element (step S3). Next, the construction unit 13 determines whether sufficient machine learning has been performed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeatedly performed while varying the parameters until sufficient machine learning has been performed. The number of iterations of machine learning varies corresponding to the quality conditions that the constructed, trained model M should meet.

[0056] After sufficient machine learning has been performed, the construction unit 13 saves the input-output relationship between the input time series data A and the output finger information B, learned through machine learning in step S3, as the trained model M (step S5). Thus, the training process ends.

[0057] Figure 6 It means by Figure 3 A flowchart illustrating an example of finger prompting processing performed by the finger prompting device 20. Figure 6 The operation prompts are processed by... Figure 1The CPU 130 executes the finger prompting program. First, the receiving unit 21 receives time-series data (step S11). Next, the estimation unit 22 uses the trained model M saved in step S5 of the training process to estimate the finger information based on the time-series data received in step S11 (step S12).

[0058] Then, the generation unit 23 generates musical score information based on the note string of time-series data received in step S11 and the finger information estimated in step S12 (step S13). The auxiliary musical score can be displayed on the display unit 160 based on the generated musical score information. Thus, the finger movement prompting process ends.

[0059] (5) Effects of the implementation method

[0060] As explained above, the finger movement prompting device 20 according to this embodiment includes: a receiving unit 21 that receives time-series data including a string of notes consisting of multiple notes; and an estimation unit 22 that uses a trained model M to estimate finger information representing the fingers used when playing each note of the string of notes on an instrument. According to this structure, the trained model M is used to estimate appropriate finger information based on the time progression of multiple notes in the time-series data. Therefore, appropriate finger movements can be prompted when playing an instrument.

[0061] A trained model M can be a machine learning model that has learned the input-output relationship between input time-series data A, which includes a reference note string consisting of multiple notes, and output finger information B, representing the fingers used when playing each note of the reference note string on an instrument. In this case, finger information can be easily inferred from the time-series data.

[0062] The time-series data also includes a performer identifier representing the performer of the string of notes, and the estimation unit 22 can estimate finger information based on the performer identifier. In this case, appropriate finger information can be estimated based on the performer.

[0063] The performer identifier can be determined in a way that corresponds to the performer's physical characteristics. In this case, appropriate finger information can be inferred based on the performer's physical characteristics.

[0064] The performer identifier can be determined in a way that corresponds to the performer's playing style. In this case, the appropriate finger information can be inferred based on the performer's playing style.

[0065] The fingering prompting device 20 may also include a generation unit that generates sheet music information representing auxiliary musical scores that assign finger information in a manner corresponding to each note in a note string. In this case, the performer can easily identify the fingers corresponding to each note in the note string by visually reviewing the auxiliary musical score.

[0066] The training device 10 according to this embodiment includes: a first acquisition unit 11 that acquires input time-series data A including a reference note string composed of multiple notes; a second acquisition unit 12 that acquires output finger information B representing the fingers used when playing each note of the reference note string on an instrument; and a construction unit 13 that constructs a trained model M that learns the input-output relationship between the input time-series data A and the output finger information B. According to this structure, a trained model M that learns the input-output relationship between the input time-series data A and the output finger information B can be easily constructed.

[0067] (6) Other examples of training data

[0068] In this embodiment, the input time series data A includes a reference performer identifier, and the time series data includes a performer identifier, but the embodiment is not limited to this. The input time series data A only needs to contain a reference note string, or it may not contain a reference performer identifier. Similarly, the time series data only needs to contain a note string, or it may not contain a performer identifier.

[0069] Furthermore, in this embodiment, the input time sequence data A and the output finger information B are recorded according to the so-called motion references in the MIDI (Musical Instrument Digital Interface) standard, which represent key presses or releases, etc., but the embodiment is not limited to this. The input time sequence data A and the output finger information B can be recorded in other ways. For example, the input time sequence data A and the output finger information B can be recorded according to the so-called note references in the MIDI standard, which represent the start position of a note or the length of a note, etc. The same applies to the time sequence data and finger information.

[0070] Figure 7 This is a graph representing other examples of input time series data A. In Figure 7 The upper section shows the input time series data A(Ax) recorded according to the action reference. Figure 7 The middle section shows the input time series data A (Ay) written according to the note reference. The input time series data Ax and Ay contain the same reference note string (in... Figure 7The lower section shows the reference note string in the musical score. The "bar" and "beat" in the input time series data Ax and Ay are elements representing the beat construction of the reference note string.

[0071] like Figure 7 As shown, by recording the input time sequence data A according to the note reference, the length of the input time sequence data A is shortened. This allows for easier processing of longer input time sequence data A. Furthermore, the output finger information B corresponding to the input time sequence data A can be recorded by inserting an element representing the finger number immediately following the element representing the pitch number (“note_○○”) in the input time sequence data A.

[0072] Alternatively, the input time series data A and the output finger information B can be represented as musical notation. Details regarding the representation of input time series data A and output finger information B as musical notation are explained in the following variation.

[0073] (7) Variation

[0074] Figure 8 This is a graph representing an example of the input time series data A in a variant example. In Figure 8 The upper section shows the input time series data A(Az) recorded in a manner resembling musical notation. Figure 8 The lower section shows the musical score represented by the input time series data A. For example... Figure 8 As shown in the previous paragraph, the input time series data Az contains multiple features A0 to A24. Some features have attributes. The attributes of a feature are described after the feature (after the underscore).

[0075] Element A0 represents the proportion of notes assigned to fingering within the reference note string. Element A0 is placed before the beginning of the input time series data Az, but it can also be placed anywhere within the input time series data Az. The proportion is specified according to the "fingerrate" attribute of element A0. In this example, the attribute "5" refers to a proportion of 100%. The proportion can have a range, such as 20–40% or 40–60%, or it can be divided into multiple ranges.

[0076] Element A1 represents a voice part. Element A1 is positioned immediately after element A0, but it can also be positioned anywhere in the input time series data Az. As element A1, "R" and "L" represent the right-hand and left-hand voice parts, respectively. In this example, the element corresponding to the right hand is positioned after "R". "L" is then positioned after it, followed by the element corresponding to the left hand. The elements corresponding to "R" and the right hand can be positioned after the elements corresponding to the left hand. Without distinguishing voice parts, the input time series data Az does not contain element A1.

[0077] Elements A2, A15, and A24 represent bar lines in musical notation. Therefore, in Figure 8 In the example, the area defined by "bar" in element A2 and "bar" in element A15 corresponds to section 1. The area defined by "bar" in element A15 and "bar" in element A24 corresponds to section 2.

[0078] Element A3 represents the musical notation for different parts of the score. The type of musical notation is specified based on the "clef" attribute within element A3. Figure 8 In the example, the attribute is "treble", so the "ト" sound symbol is assigned as the syllable symbol based on element A3. Furthermore, in the case of the attribute "bass", the "ヘ" sound symbol is assigned as the syllable symbol based on element A3.

[0079] Element A4 represents the time signature in the musical score. The type of time signature is specified based on the "time" attribute within element A4. Figure 8 In the example, the attribute is "4 / 4", so "4 / 4" is specified as the time signature based on element A4.

[0080] Notes in a reference note string are represented by a combination of pitch and note value. Pitch is specified by the "note" attribute in elements A5, A9, A11, A13, A16, A18, and A20. Note value is specified by the "len" attribute in elements A6, A10, A12, A14, A17, A19, and A21. In this example, "len_1" is equivalent to 1 beat.

[0081] The direction of the note stems in a musical score is specified by other attributes of "len" in elements A6, A10, A12, A14, A17, A19, and A21. When other attributes are "down", the stem extends downwards from the notehead. When other attributes are "up", the stem extends upwards from the notehead. When multiple notes, such as eighth notes or sixteenth notes, are connected by beams, the start, middle, and end positions of the beams are specified by "start", "continue", and "stop" attributes of "len" in elements A10, A12, and A14, respectively.

[0082] The rests in a string of notes are specified by the "rest" attribute in elements A7 and A22. The note value of a rest is described by the "len" attribute in elements A8 and A23.

[0083] exist Figure 8 In the example, elements A5 and A6 represent note N1, and elements A7 and A8 represent rest R1. Elements A9 and A10 represent note N2, elements A11 and A12 represent note N3, and elements A13 and A14 represent note N4. Elements A16 and A17 represent note N5, and elements A18 and A19 represent note N6. Elements A20 and A21 represent note N7, and elements A22 and A23 represent rest R2.

[0084] Figure 9 This is a diagram illustrating an example of the output finger information B in a modified example. Figure 9 The upper section shows the output finger information B(Bz) recorded in a way that represents musical notation. The output finger information Bz and... Figure 8 The input time series data Az corresponds to this. Figure 9 The lower section shows the musical score represented by the output finger information Bz.

[0085] like Figure 9 As shown in the upper section, the output finger information Bz contains multiple elements B0 to B24. Additionally, the output finger information Bz also includes elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f, respectively, positioned immediately after elements B5, B9, B11, B13, B16, B18, and B20. Elements B0 to B24 are respectively associated with… Figure 8 The input time series data Az has the same elements A0 to A24. Therefore, Figure 3 The first acquisition unit 11 can acquire the input time series data Az by deleting elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz.

[0086] Elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f represent the finger numbers used when playing the notes corresponding to the elements B5, B9, B11, B13, B16, B18, and B20 immediately preceding these elements on an instrument. The finger numbers are specified by the "finger" attribute of elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. Therefore, based on elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f, as... Figure 9 As shown in the lower section, the finger numbers “1”, “1”, “2”, “1”, “3”, “3” and “2” used when playing notes N1 to N7 are respectively recorded in the musical score.

[0087] (8) Effects of the variation

[0088] In a variation of the first embodiment, the proportion of notes assigned to fingering in the reference note string can be arbitrarily specified based on the attributes of element A0. When the proportion is 100%, the estimation unit 22 estimates finger information for all notes in the note string. In this case, it is possible to prompt beginner-level players with appropriate fingering techniques when playing the instrument.

[0089] Furthermore, when the scale is 100%, the generation unit 23 can generate an animation file representing finger movements through animation or other means based on the finger information estimated by the estimation unit 22. This makes the finger movements visible. The generation of this animation file can... Figure 6 The operation prompt can be executed before or after step S13, can be executed in parallel with step S13, or can replace step S13.

[0090] On the other hand, when the proportion is less than 100%, the estimation unit 22 estimates a portion of the notes in the note string that are the objects of fingering, as well as finger information about that portion of the notes. In this case, it can provide appropriate fingering suggestions to beginner or intermediate level players who are above the beginner level when playing the instrument. In this structure, the output finger information Bz does not include a portion of elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f.

[0091] Furthermore, when the ratio is less than 100%, the estimation unit 22 may not estimate the finger information, but instead estimate the note information of the note representing the object to be fingered, based on the note string. Details will be explained in the third embodiment described later.

[0092] [2] Second Embodiment

[0093] (1) Processing system

[0094] Regarding the processing system 100 of the second embodiment, the differences from the processing system 100 of the first embodiment will be described. Figure 3 In the training device 10, the first acquisition unit 11 and the second acquisition unit 12 acquire the input time series data A and the output finger information B of the training data D, respectively.

[0095] Figure 10 This is a diagram illustrating an example of input time series data A in the second embodiment. Figure 10 The upper section shows the input time series data Az, recorded in a manner resembling musical notation. Figure 10 The lower section shows the musical score represented by the input time series data Az.

[0096] like Figure 10 As shown in the upper paragraph, the input time series data Az contains multiple elements A0 to A24. Figure 10 Elements A0 to A24 are respectively modified from the first embodiment ( Figure 8 The elements A0 to A24 are the same. Additionally, the input time series data Az contains additional elements positioned immediately following a portion of the elements A5, A9, A11, A13, A16, A18, and A20 corresponding to the notes. Figure 10 In the example, the input time series data Az also includes elements A5f, A11f, A16f, and A20f, which are configured immediately after the elements A5, A11, A16, and A20, respectively.

[0097] Elements A5f, A11f, A16f, and A20f are finger information (hereinafter referred to as basic finger information) representing the finger numbers used when playing the notes corresponding to the elements A5, A11, A16, and A20 immediately preceding the aforementioned elements on an instrument. The finger number is specified by the "finger" attribute in elements A5f, A11f, A16f, and A20f. Therefore, based on elements A5f, A11f, A16f, and A20f, as... Figure 10 As shown in the lower section, the finger numbers “1”, “2”, “3” and “2” used when playing notes N1, N3, N5 and N7 are respectively recorded in the musical score.

[0098] The output finger information Bz of this embodiment is a variation of the first embodiment ( Figure 9The output finger information Bz is the same. Therefore, the first acquisition unit 11 can acquire the input time series data Az by randomly deleting a portion of elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz. The proportion of deleted elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f can be determined by the user of the training device 10. Figure 1 The operation is specified by the operation unit 150.

[0099] In this example, the input time series data Az is obtained by deleting elements B9f, B13f, and B18f from the output finger information Bz. The elements B5f, B11f, B16f, and B20f that were not deleted remain as the basic finger information, i.e., elements A5f, A11f, A16f, and A20f.

[0100] Figure 3 The construction unit 13 performs machine learning by setting the input time series data Az as the input element and the output finger information Bz as the output element. By repeatedly performing machine learning on multiple training data D, a trained model M representing the input-output relationship between the input time series data Az and the output finger information Bz is constructed.

[0101] In the fingering prompting device 20, the receiving unit 21 receives time-series data. The time-series data also includes basic finger information representing the fingers used when playing a portion of the notes in a note string via an instrument. The estimation unit 22 estimates the finger information representing the fingers used when playing the notes in a note string via an instrument based on the constructed and trained model M and the basic finger information. The generation unit 23 generates musical score information based on the note string and finger information in the time-series data.

[0102] (2) Effects of the implementation method

[0103] According to this embodiment, even when only finger information (basic finger information) for a portion of the notes in a note string of time-series data is known, and finger information for the remaining notes is not provided, finger information for the remaining notes can still be supplemented. Therefore, beginner-level players can be prompted with appropriate fingering when playing the instrument. The generation unit 23 can generate an animation file representing finger movements through animation or the like based on the finger information estimated by the estimation unit 22. In this case, the finger movements can be made visible.

[0104] (3) Variations

[0105] In this embodiment, the estimation unit 22 estimates finger information for all notes contained in the note string of time-series data, but the embodiment is not limited to this. When finger information is given for a first proportion of the notes in the note string, the estimation unit 22 can estimate finger information for a second proportion of the notes in the note string, where the second proportion is greater than the first proportion. In this case, beginner or intermediate level players can be prompted with appropriate fingering when playing the instrument.

[0106] In a variation, the output finger information B of the training data D may not include a portion of elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. For example, if the input time series data Az contains elements A5f, A11f, A16f, and A20f, the output finger information B may contain elements B5f, B11f, B16f, and B20f. On the other hand, the output finger information B may not include a portion of elements B9f, B13f, and B18f.

[0107] [3] Third embodiment

[0108] (1) Processing system

[0109] Regarding the processing system 100 of the third embodiment, the differences from the processing system 100 of the first embodiment will be explained. In this embodiment, the training data D represents a group of input time series data A and output note information. Figure 3 In the training device 10, the first acquisition unit 11 and the second acquisition unit 12 acquire the input time series data A and output note information of the training data D, respectively. The acquisition of the output note information is replaced by... Figure 5 This is performed in step S2 of the sound learning process.

[0110] The input time series data Az in this embodiment is different from the variant of the first embodiment ( Figure 8 The input time series data Az is the same. The first acquisition unit 11 can obtain the data from the data described later. Figure 11 The output note information Cz is used to delete elements C9f, C11f, and C16f to obtain the input time series data Az.

[0111] Figure 11 This is a diagram illustrating an example of the output note information C in the third embodiment. Figure 11 The upper section shows the output note information C (Cz) as represented by musical notation. Figure 11 The lower section shows the musical score represented by the output note information Cz.

[0112] like Figure 11As shown in the upper section, the output note information Cz contains multiple elements C0 to C24. Figure 11 Elements C0 to C24 are respectively modified from the first embodiment ( Figure 9 The output finger information Bz has the same elements B0 to B24. In addition, the output note information Cz includes additional elements that are configured immediately after a portion of the elements C5, C9, C11, C13, C16, C18, and C20 corresponding to the note.

[0113] In this example, the "fingerrate" attribute of element C0 is "2", which refers to a proportion of 40%. Therefore, the output note information Cz also includes elements C9f, C11f, and C16f, which are configured after approximately 40% of the elements immediately adjacent to elements C5, C9, C11, C13, C16, C18, and C20, respectively.

[0114] Elements C9f, C11f, and C16f, along with the notes corresponding to the elements C9, C11, and C16 immediately preceding them, are shown as notes that become the objects of fingering assignment from the reference note string. For example... Figure 11 As shown in the lower section, based on elements C9f, C11f, and C16f, the notes N2, N3, and N5 corresponding to elements C9, C11, and C16 are recorded in the musical score in a recognizable manner.

[0115] Figure 3 The construction unit 13 performs machine learning by setting the input time series data Az as the input element and the output note information Cz as the output element. By repeatedly performing machine learning on multiple training data D, a trained model M representing the input-output relationship between the input time series data Az and the output note information Cz is constructed.

[0116] In the fingering prompting device 20, the receiving unit 21 receives time-series data. The estimation unit 22, based on the trained model M constructed by the training device 10 and the time-series data received by the receiving unit 21, estimates note information representing the notes to be fingered from the note string. The estimation of note information replaces... Figure 6 The step S12, which involves processing finger placement prompts, is executed. The generation unit 23 generates musical score information representing an auxiliary musical score, which displays the notes represented by the note information in a recognizable manner.

[0117] (2) Effects of the implementation method

[0118] According to this embodiment, it is possible to indicate the notes that are the objects of fingering from the note string. Thus, beginner or intermediate level players can identify the key notes when playing the instrument.

[0119] (3) Variations

[0120] The estimation unit 22 can use the first trained model M constructed in the first embodiment and the second trained model M constructed in this embodiment to estimate the finger information representing the finger used when playing a portion of the notes contained in a string of notes through an instrument. Figure 12 This is a flowchart illustrating an example of pointer hint processing for a variant.

[0121] First, the receiving unit 21 receives time-series data (step S21). Next, the estimation unit 22 uses the first trained model M constructed in the first embodiment to estimate the intermediate finger information based on the time-series data received in step S11 (step S22). The intermediate finger information represents the finger used when playing each note in a string of notes on an instrument.

[0122] In addition, the estimation unit 22 uses the second trained model M constructed in this embodiment to estimate the note information based on the time series data received in step S11 (step S23). Regarding steps S22 and S23, either one can be executed first, or they can be executed simultaneously.

[0123] Next, the estimation unit 22, based on the middle finger information estimated in step S22, estimates the finger information for notes in the note string other than those represented by the note information estimated in step S23 (step S24). Then, the generation unit 23 generates musical score information based on the note string of time-series data received in step S21 and the finger information estimated in step S24 (step S25). Thus, the finger prompting process ends.

[0124] In this finger gesture prompting process, the information of the middle finger estimated in step S22 has, for example, a variation of the first embodiment ( Figure 9 The output finger information Bz has the same structure. Furthermore, the note information estimated in step S23 has the same... Figure 11 The output note information Cz has the same structure. Figure 13 This is a diagram illustrating an example of finger information estimated in step S24 of the finger prompting process.

[0125] exist Figure 13 The upper section shows finger information F (Fz) recorded in a way that represents musical notation. Figure 13 The lower section shows an auxiliary musical score represented by finger information Fz. Finger information Fz is derived from the middle finger information (see reference). Figure 9 ) Delete the information representing the note (see reference) Figure 11The elements C9f, C11f, and C16f that are the notes to be assigned to the fingering are presumed to be B9f, B11f, and B16f, respectively.

[0126] Specifically, such as Figure 13 As shown in the upper paragraph, the finger information Fz contains multiple elements F1 to F24. Figure 13 Elements F1 to F24 are respectively modified from the first embodiment ( Figure 9 The output finger information Bz has the same elements B1 to B24. Additionally, finger information Fz includes additional elements placed immediately after a portion of the elements F5, F9, F11, F13, F16, F18, and F20 corresponding to the notes. In this example, finger information Fz also includes elements F5f, F13f, F18f, and F20f placed immediately after elements F5, F13, F18, and F20, respectively.

[0127] Elements F5f, F13f, F18f, and F20f represent the finger numbers used when playing the notes corresponding to the elements F5, F13, F18, and F20 immediately preceding them on an instrument. The finger numbers are specified by the "finger" attribute of elements F5f, F13f, F18f, and F20f. Therefore, based on elements F5f, F13f, F18f, and F20f, as... Figure 13 As shown in the lower section, the finger numbers “1”, “1”, “3” and “2” used when playing notes N1, N4, N6 and N7 are respectively recorded in the auxiliary musical score.

[0128] According to a variation, finger information for a subset of notes is removed from the finger information of all notes contained in a note sequence based on time-series data. In this case, beginner or intermediate level players can be prompted to use appropriate fingering when playing the instrument. For example, by removing finger information for notes that are crucial when playing the instrument, beginner or intermediate level players can develop the judgment for appropriate fingering while practicing the instrument.

[0129] [4] Other implementation methods

[0130] In the above embodiment, the finger movement prompting device 20 includes a generation unit 23, but the embodiment is not limited to this. The performer can transfer the finger information estimated by the estimation unit 22 to the desired score to create an auxiliary score. Therefore, the finger movement prompting device 20 may not include a generation unit 23.

[0131] In the above embodiment, the training data D is trained to estimate finger information when playing a piano, but the embodiment is not limited to this. The training data D can be trained to estimate finger information when playing other instruments such as drums.

[0132] In the above embodiment, the example described is that the user of the finger movement prompting device 20 is a performer, but the user of the finger movement prompting device 20 could also be an employee of a sheet music production company. Furthermore, the machine learning training of the training device 10 could be performed in advance by an employee of the sheet music production company.

Claims

1. A finger movement prompting device, comprising: The receiving unit accepts time-series data including strings of notes consisting of multiple notes; and The estimation unit uses a trained model to estimate finger information or note information, where the finger information represents the finger used when playing at least a portion of the notes contained in the note string through an instrument, and the note information represents the note that becomes the object of fingering from the note string. The time-series data also includes a performer identifier representing the performer playing the note string, the identifier being determined in a manner that varies based on at least one of the performer's physical characteristics and the performer's playing style. The estimation unit estimates the finger information based on the performer identifier.

2. The finger movement prompting device according to claim 1, wherein, The trained model is a machine learning model that learns the input-output relationship between input time series data and output finger information or output note information. The input time series data includes a reference note string consisting of multiple notes. The output finger information represents the finger used when playing at least a portion of the notes contained in the reference note string through an instrument. The output note information represents the note that becomes the object of fingering from the reference note string.

3. The finger movement prompting device according to claim 1 or 2, wherein, The time-series data also includes basic finger information, which indicates the fingers used when playing a first proportion of the notes in the note sequence using an instrument. The estimation unit estimates the finger information representing the finger used when playing notes in a second proportion of the notes contained in the note string through an instrument, based on the basic finger information, where the second proportion is greater than the first proportion.

4. The finger movement prompting device according to claim 1 or 2, wherein, The estimation unit estimates the middle finger information and the note information, which indicate the fingers used when playing each note in the note string through a musical instrument. Based on the middle finger information, it estimates the finger information, which indicates the fingers used when playing notes other than those indicated by the note information, in the note string through a musical instrument.

5. The finger movement prompting device according to claim 1, wherein, It also has a generation unit that generates musical score information, which represents a first auxiliary musical score that assigns the finger information in a manner corresponding to at least a portion of the notes contained in the note string, or a second auxiliary musical score that displays the notes represented by the note information in a recognizable manner.

6. A training device comprising: The first acquisition unit acquires input time series data, including a reference note string consisting of multiple notes; The second acquisition unit acquires output finger information or output note information, wherein the output finger information indicates the finger used when playing at least a portion of the notes contained in the reference note string via an instrument, and the output note information indicates the note to which the fingering is assigned from the reference note string; and The construction unit builds a trained model that learns the input-output relationship between the input time-series data and the output finger information or the output note information. The time-series data also includes a performer identifier representing the performer playing the note string, the performer identifier being determined in a manner that varies based on at least one of the performer's physical characteristics and the performer's playing style.

7. A finger gesture prompting method, which is executed by a computer. It accepts time-series data containing strings of notes consisting of multiple notes. The trained model is used to estimate finger information or note information, where the finger information represents the finger used when playing at least a portion of the notes contained in the note string through an instrument, and the note information represents the note that becomes the object of fingering from the note string. The time-series data also includes a performer identifier representing the performer playing the note string, the performer identifier being determined in a manner that varies based on at least one of the performer's physical characteristics and the performer's playing style. The finger information is inferred based on the performer identifier.

8. The finger gesture prompting method according to claim 7, wherein, The trained model is a machine learning model that learns the input-output relationship between input time series data and output finger information or output note information. The input time series data includes a reference note string consisting of multiple notes. The output finger information represents the finger used when playing at least a portion of the notes contained in the reference note string through an instrument. The output note information represents the note that becomes the object of fingering from the reference note string.

9. The finger gesture prompting method according to claim 7 or 8, wherein, The time-series data also includes basic finger information, which indicates the fingers used when playing a first proportion of the notes in the note sequence using an instrument. Based on the basic finger information, the finger information representing the fingers used when playing notes in a second proportion of the notes contained in the note string through an instrument is estimated, the second proportion being greater than the first proportion.

10. The finger gesture prompting method according to claim 7 or 8, wherein, The information of the middle finger, representing the finger used when playing each note in the string of notes by means of an instrument, and the information of the note are estimated. Based on the information of the middle finger, the information of the finger used when playing notes in the string of notes other than the notes represented by the information of the note are estimated.

11. The finger gesture prompting method according to claim 7, wherein, It also generates musical score information, which represents a first auxiliary musical score that assigns the finger information in a manner corresponding to at least a portion of the notes contained in the note string, or a second auxiliary musical score that displays the notes represented by the note information in a recognizable manner.

12. A training method performed by a computer, Obtain the input time series data, which includes a reference note string consisting of multiple notes. Obtain output finger information or output note information, wherein the output finger information indicates the finger used when playing at least a portion of the notes contained in the reference note string through an instrument, and the output note information indicates the note that becomes the object of fingering from the reference note string. A trained model is constructed that learns the input-output relationship between the input time-series data and the output finger information or the output note information. The time-series data also includes a performer identifier representing the performer playing the note string, the performer identifier being determined in a manner that varies based on at least one of the performer's physical characteristics and the performer's playing style.

Citation Information

Patent Citations

  • Fingering determination method and system for musical instrument performance

    JP2007241034A

  • Device, method, and program for processing information

    JP2013083845A