Code estimation device and code estimation method

Abstract

To provide a code estimation device and method for performing highly flexible chord estimation based on a sequence of musical notes for playing musical instruments.SOLUTION: In a processing system 100 including a chord estimation device 20 for estimating chords for playing a musical instrument and a training device 10, the estimation device comprises: a receiving unit 21 that accepts time-series data containing a sequence of musical notes comprising multiple notes, based on an image of a musical score or an operation using an operation unit 150 operated by a performer; an estimation unit 22 that estimates chord sequence information corresponding to the sequence of musical notes and recording chord change timing, based on the time-series data; and a generation unit 23 that generates musical score information based on the sequence of musical notes of the time-series data accepted by the receiving unit 21 and the chord sequence information estimated by the estimation unit 22.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a chord estimator and method for estimating chords for playing a musical instrument, and a training device and method for building a chord estimator. [Background technology]

[0002] There are musical scores with chords added. Performers can enjoy playing musical instruments by playing the chords on instruments such as piano and guitar. When creating musical scores with chords, the creator adds chords based on the melody and accompaniment sounds indicated by the notes. The task of adding chords requires musical knowledge and sense. Patent Document 1 listed below discloses a chord progression estimation and detection device that estimates chords from performance information or audio signals.

[0003] [Patent Document 1] Patent No. 6151121 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, chords are estimated for each specific section. For example, one chord is estimated for each measure. If chord estimation with a higher degree of freedom could be performed from given notes, it is expected that more appropriate support could be provided for the creation of sheet music with chords.

[0005] An object of the present invention is to perform highly flexible chord estimation based on a sequence of notes. [Means for solving the problem]

[0006] A chord estimation device according to one aspect of the present invention includes a receiving unit that receives time-series data including an image of a musical score or a note sequence consisting of a plurality of notes via an operation using an operation unit, and an estimating unit that estimates chord sequence information indicating a chord sequence corresponding to the note sequence, with chord change timings recorded, based on the time-series data.

[0007] A chord estimating device according to another aspect of the present invention includes a receiving unit that receives time-series data including a note sequence consisting of a plurality of notes through an operation using an operation unit, and an estimating unit that estimates chord sequence information, based on the time-series data, that indicates a chord sequence corresponding to the note sequence and in which chord change timings are recorded.

[0008] A computer-executable chord estimation method according to yet another aspect of the present invention accepts time-series data including a note sequence consisting of a plurality of notes, either via an image of a musical score or via operation using an operation unit, and estimates, based on the time-series data, chord sequence information that indicates a chord sequence corresponding to the note sequence and in which chord change timings are recorded.

[0009] A computer-executed chord estimation method according to yet another aspect of the present invention receives time-series data including a note sequence consisting of a plurality of notes via an operation using an operation unit, and estimates, based on the time-series data, chord sequence information that indicates a chord sequence corresponding to the note sequence and in which chord change timings are recorded. [Effects of the Invention]

[0010] According to the present invention, it is possible to perform chord estimation with a high degree of freedom based on a sequence of notes. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of a processing system including a chord estimation device and a training device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of input time-series data included in the training data. [Figure 3] FIG. 3 is a diagram showing an example of output code string information included in the training data. [Figure 4] FIG. 4 is a block diagram showing the configuration of the training device and the chord estimation device. [Figure 5] FIG. 5 shows an example of an arranged musical score displayed on the display unit. [Figure 6] FIG. 6 is a flowchart showing an example of the training process. [Figure 7] FIG. 7 is a flowchart showing an example of the chord estimation process. [Figure 8] FIG. 8 is a diagram showing a modified example of the output code string information included in the training data. DETAILED DESCRIPTION OF THE INVENTION

[0012] (1) Processing system configuration A chord estimation device, a training device, a chord estimation method, and a training method according to embodiments of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a chord estimation device and a training device according to one embodiment of the present invention. As shown in Fig. 1, processing system 100 includes RAM (random access memory) 110, ROM (read only memory) 120, CPU (central processing unit) 130, storage unit 140, operation unit 150, and display unit 160.

[0013] The processing system 100 is realized by a computer such as a personal computer, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet, or may be realized by an electronic musical instrument with a performance function such as an electronic piano.

[0014] The RAM 110, ROM 120, CPU 130, storage unit 140, operation unit 150, and display unit 160 are connected to a bus 170. The RAM 110, ROM 120, and CPU 130 constitute the training device 10 and the chord estimation device 20. In this embodiment, the training device 10 and the chord estimation device 20 are configured by a common processing system 100, but they may also be configured by separate processing systems.

[0015] The RAM 110 is made of, for example, a volatile memory and is used as a work area for the CPU 130. The ROM 120 is made of, for example, a nonvolatile memory and stores a training program and a chord estimation program. The CPU 130 performs training processing by executing the training program stored in the ROM 120 on the RAM 110. The CPU 130 also performs chord estimation processing by executing the chord estimation program stored in the ROM 120 on the RAM 110. Details of the training processing and chord estimation processing will be described later.

[0016] The training program or the chord estimation program may be stored in the storage unit 140 instead of the ROM 120. Alternatively, the training program or the chord estimation program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the training program or the chord estimation program may be distributed from a server (including a cloud server) on the network and installed in the ROM 120 or the storage unit 140.

[0017] The storage unit 140 includes a storage medium such as a hard disk, an optical disk, a magnetic disk, or a memory card, and stores the trained model M and a plurality of pieces of training data D. The trained model M or each piece of training data D may be stored in a computer-readable storage medium instead of being stored in the storage unit 140. Alternatively, if the processing system 100 is connected to a network, the trained model M or each piece of training data D may be stored in a server on the network.

[0018] (2) Training data The trained model M is a machine learning model trained to present chord sequences that a user of the chord estimation device 20 (hereinafter referred to as a performer) can refer to when playing a piece of music. The trained model M is constructed using multiple sets of training data D. A user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is data created based on the musical knowledge or musical sense of a reference performer. The reference performer has a relatively high level of skill in playing a piece of music. The reference performer may also be the performer's instructor or teacher in playing a piece of music.

[0019] The training data D represents a set of input time series data and output chord sequence information. The input time series data represents a reference note sequence consisting of multiple notes. For example, the input time series data is data that comprises multiple notes forming a melody or accompaniment. The input time series data may be image data representing an image of a musical score. The output chord sequence information is data in which chords corresponding to the reference note sequence are arranged in time series. The chord sequence corresponding to the reference note sequence is assigned by a reference performer.

[0020] 2 and 3 are diagrams showing examples of each training data D. The example in Fig. 2 shows input time-series data including a reference note sequence consisting of multiple notes. The example in Fig. 3 shows output chord sequence information indicating a chord sequence corresponding to the reference note sequence.

[0021] In this embodiment, the input time series data has a metrical structure and additional information in addition to a reference note sequence. Input time series data A shown in FIG. 2 is data obtained by extracting data for the first two bars of a song. In input time series data A, bars are separated by "bars" and beats are separated by "beats." In this way, input time series data A has a metrical structure based on the "bar" and "beat" information. Elements A1 to A37 indicate the reference note sequence of the first bar. In other words, elements A1 to A37 are separated into bars by the "bar" before element A1 and the "bar" after element A37. In addition, elements A8, A18, and A26 are separated into beats by the "beats" after elements A8, A18, and A26.

[0022] Element A0 is additional information. Examples of additional information that can be used include key information, genre information, and difficulty level information. In the example of FIG. 2, key information is added using the Key element. Key information is information that specifies the key of the music represented by the reference note sequence. The number following Key specifies the key. By specifying key information as additional information, a chord sequence corresponding to the reference note sequence and the key is machine-learned. Genre information is information that specifies the genre of the music represented by the reference note sequence. Examples of genre information that can be specified include rock, pop, and jazz. By specifying genre information as additional information, a chord sequence corresponding to the reference note sequence and the genre is machine-learned. Difficulty information is information that indicates the difficulty level of the musical score represented by the reference note sequence. By specifying difficulty information as additional information, a chord sequence corresponding to the reference note sequence and the difficulty level of the musical score is machine-learned. For example, for a low-difficulty score, machine learning is performed by interpolating notes from a small number of notes. On the other hand, for a high-difficulty score, machine learning is performed by selecting notes that constitute chords from an excessive number of notes.

[0023] Of the elements of the input time series data A, elements other than element A0, "bar," and "beat" correspond to the reference note sequence. Elements A1 to A37 indicate the reference note sequence of the first bar. In this example, element A0 is placed at the beginning of the input time series data A, that is, before the reference note sequence (elements A1 to A37), but it may be placed at any position in the input time series data A.

[0024] As exemplified by elements A1 to A37, in the reference note sequence, "L" represents the left hand, "R" represents the right hand, and the number following "L" or "R" represents the scale. Also, "on" and "off" represent key pressing and key release, respectively. Also, "wait" represents waiting, and the number following "wait" represents the length of time. Therefore, elements A1 to A5 indicate that the right hand presses keys 77 and 74, while the left hand simultaneously presses keys 53 and 46, and then maintains these positions for 11 time units. After maintaining these positions for 11 time units, elements A6 to A8 indicate that the left hand simultaneously releases keys 53 and 46, and then maintains these positions for one time unit. After maintaining these positions for one time unit, elements A9 to A11 indicate that the left hand presses keys 53 and 46 again, and then waits for five time units.

[0025] The output chord sequence information B shown in Figure 3 indicates a chord sequence corresponding to the reference note sequence included in the input time series data A. The chord sequence corresponding to elements A1 to A37 of the input time series data A is represented by elements B1 to B3 and elements B4 to B6. In other words, elements B1 to B6 indicate a chord sequence corresponding to the first bar of the input time series data A. In the output chord sequence information B, bars are also separated by "bar" and beats are separated by "beat." The range separated by the "bar" before element B1 and the "bar" after element B6 corresponds to the first bar.

[0026] In the output chord sequence information B, one chord is represented by three elements. Elements B1 to B3 specify the chord on the first beat of the first measure. Elements B4 to B6 specify the chord on the fourth beat of the first measure. Elements B7 to B9 specify the chord on the first beat of the fourth measure. Of the three elements indicating the chord, the first element (B1, B4, B7) indicates the basic chord information. The basic chord information (chord) indicates a number from 1 to 24 that specifies the type of major chord or minor chord for each of the 12 notes (C, C#, D, D#, A, A#, B). Of the three elements indicating the chord, the second element (B2, B5, B8) indicates the chord type information. The chord type information (type) indicates a number that specifies the type of tension chord. Of the three elements indicating the chord, the third element (B3, B6, B9) indicates the chord root information. The chord root information (root) indicates a number that specifies the root note of the on-chord.

[0027] (3) Training equipment and code estimation equipment Fig. 4 is a block diagram showing the configurations of the training device 10 and the chord estimation device 20. As shown in Fig. 4, the training device 10 includes, as functional units, a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13. The functional units of the training device 10 are realized by the CPU 130 in Fig. 1 executing a training program. At least some of the functional units of the training device 10 may be realized by hardware such as electronic circuits.

[0028] First acquisition unit 11 acquires input time-series data A from each piece of training data D stored in storage unit 140 or the like. Second acquisition unit 12 acquires output code string information B from each piece of training data D. Construction unit 13 performs machine learning for each piece of training data D, using the input time-series data A acquired by first acquisition unit 11 as an input element and the output code string information B acquired by second acquisition unit 12 as an output element. By repeating machine learning for multiple pieces of training data D, construction unit 13 constructs a trained model M that indicates the input-output relationship between the input time-series data A and the output code string information B.

[0029] In this example, the construction unit 13 constructs the trained model M by training a Transformer, but the embodiment is not limited to this. The construction unit 13 may construct the trained model M by training another type of machine learning model that handles time series. The trained model M constructed by the construction unit 13 is stored in, for example, the storage unit 140. The trained model M constructed by the construction unit 13 may also be stored in a server on a network, etc.

[0030] The chord estimation device 20 includes, as functional units, a receiving unit 21, an estimation unit 22, and a generation unit 23. The functional units of the chord estimation device 20 are realized by the CPU 130 in Fig. 1 executing a chord estimation program. At least some of the functional units of the chord estimation device 20 may be realized by hardware such as an electronic circuit.

[0031] In this embodiment, the receiving unit 21 receives time-series data including a sequence of notes made up of a plurality of notes. The performer can provide image data showing an image of a musical score to the receiving unit 21 as time-series data. Alternatively, the performer can generate time-series data by operating the operation unit 150 and provide the time-series data to the receiving unit 21. In this example, the time-series data has the same structure as the input time-series data A in FIG. 2. That is, the time-series data includes a metrical structure and additional information in addition to a sequence of notes.

[0032] The estimation unit 22 estimates chord sequence information using a trained model M stored in the storage unit 140 or the like. The chord sequence information indicates a chord sequence corresponding to the note sequence accepted by the acceptance unit 21, and is estimated based on the note sequence and additional information. Since the time-series data has a structure similar to that of the input time-series data A, the chord sequence information has a structure similar to that of the output chord sequence information B. The generation unit 23 generates musical score information based on the note sequence of the time-series data accepted by the acceptance unit 21 and the chord sequence information estimated by the estimation unit 22. For example, the musical score information is information on an arranged piano score, and is data in which chord information is added to a staff. Alternatively, the musical score information is MIDI data to which chord sequence information has been added.

[0033] The display unit 160 displays a score with chords based on the musical score information generated by the generation unit 23. Fig. 5 shows an example of a score with chords displayed on the display unit 160. As shown in Fig. 5, the score with chords displays the chord sequence information estimated by the estimation unit 22 in correspondence with each note of the sequence of notes accepted by the acceptance unit 21.

[0034] (4) Training and chord estimation Fig. 6 is a flowchart showing an example of training processing by the training device 10 of Fig. 4. The training processing of Fig. 6 is performed by the CPU 130 of Fig. 1 executing a training program. First, the first acquisition unit 11 acquires input time-series data A from each training data D (step S1). Furthermore, the second acquisition unit 12 acquires output code string information B from each training data D (step S2). Steps S1 and S2 may be executed either first or simultaneously.

[0035] Next, the construction unit 13 performs machine learning for each training data D, using the input time-series data A acquired in step S1 as an input element and the output code string information B acquired in step S2 as an output element (step S3). Subsequently, the construction unit 13 determines whether sufficient machine learning has been performed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeated while changing the parameters until sufficient machine learning has been performed. The number of times the machine learning is repeated varies depending on the quality conditions that the trained model M to be constructed must satisfy.

[0036] When sufficient machine learning has been performed, the construction unit 13 saves the input / output relationship between the input time-series data A and the output code string information B learned by the machine learning in step S3 as a trained model M (step S5), thereby completing the training process.

[0037] FIG. 7 is a flowchart showing an example of chord estimation processing by the chord estimation device 20 of FIG. 4. The chord estimation processing of FIG. 7 is performed by the CPU 130 of FIG. 1 executing a chord estimation program. First, the receiving unit 21 receives time-series data (step S11). Next, the estimation unit 22 estimates chord sequence information from the time-series data received in step S11 using the trained model M saved in step S5 of the training process (step S12). At this time, chord sequence information including one or more chord sequences is estimated from the sequence of notes included in the time-series data, allowing for highly flexible chord estimation. Furthermore, the timing of chord changes over time is also estimated, allowing for more appropriate chord estimation. In other words, although the time-series data does not include information indicating the boundaries of chord changes, the estimation unit 22 performs chord estimation that includes the timing of chord changes.

[0038] Thereafter, the generation unit 23 generates musical score information based on the musical note sequence of the time-series data accepted in step S11 and the chord sequence information estimated in step S12 (step S13). Based on the generated musical score information, a musical score with chords may be displayed on the display unit 160. This completes the chord estimation process.

[0039] (5) Effects of the embodiment As described above, the chord estimation device 20 according to this embodiment includes a receiving unit 21 that receives time-series data including a note sequence consisting of multiple notes, and an estimation unit 22 that uses a trained model M to estimate chord sequence information indicating a chord sequence corresponding to the note sequence. With this configuration, the trained model M is used to estimate appropriate chord sequence information from the temporal flow of multiple notes in the time-series data. This makes it possible to present a musical score with chords based on time-series data including a note sequence. Since one or more chord sequences are estimated from a note sequence, chord estimation can be performed with a high degree of flexibility.

[0040] The trained model M may be a machine learning model that has learned the input / output relationship between input time-series data A including a reference note sequence consisting of multiple notes and output chord sequence information B indicating chord sequences corresponding to each note in the reference note sequence. In this case, the chord sequence information can be easily estimated from the time-series data.

[0041] The estimation unit 22 may also estimate the timing of chord changes in the chord string, thereby performing more appropriate chord estimation corresponding to the note string.

[0042] The input time-series data A may include genre information that specifies the genre of the music represented by the reference note sequence. The time-series data may also include genre information that specifies the genre of the music represented by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time-series data that includes genre information. This allows chord estimation to be performed that is appropriate for the music genre.

[0043] The input time-series data A may include key information specifying the key of the music expressed by the reference note sequence. The time-series data may also include key information specifying the key of the music expressed by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time-series data including key information. This allows chord estimation to be performed that is appropriate for the key of the music.

[0044] The input time-series data A may include difficulty level information that specifies the difficulty level of the musical score represented by the reference note sequence. The time-series data may also include difficulty level information that specifies the difficulty level of the musical score represented by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time-series data that includes the difficulty level information. This allows appropriate chord estimation to be performed according to the difficulty level of the musical score represented by the note sequence.

[0045] The chord estimation device 20 may further include a generation unit 23 that generates musical score information indicating a chord-attached musical score to which chord sequence information is attached so as to correspond to each note of the musical note sequence.

[0046] A training device 10 according to this embodiment includes a first acquisition unit 11 that acquires input time-series data A including a reference note sequence consisting of a plurality of notes, a second acquisition unit 12 that acquires output chord sequence information B indicating a chord sequence corresponding to the reference note sequence, and a construction unit 13 that constructs a trained model M that has learned the input-output relationship between the input time-series data A and the output chord sequence information B. With this configuration, a trained model M that has learned the input-output relationship between the input time-series data A and the output chord sequence information B can be easily constructed.

[0047] (6) Other embodiments In the above embodiment, the input time series data A includes additional information, and the time series data includes additional information, but the embodiment is not limited to this. The input time series data A only needs to include a reference sequence of notes, and does not need to include additional information. Similarly, the time series data only needs to include a sequence of notes, and does not need to include additional information.

[0048] In the above embodiment, the input time series data A has "bar" and "beat" information as a metrical structure, but the embodiment is not limited to this. The input time series data A does not have to have a metrical structure. FIG. 8 is a diagram showing an example of output chord sequence information B prepared in response to input time series data A that does not have a metrical structure. As shown in FIG. 8, the output chord sequence information B does not have a metrical structure consisting of "bar" and "beat" information.

[0049] In the above embodiment, the input time series data A has tone information, genre information, and difficulty level information as additional information. The construction unit 13 may construct different trained models M according to the type of additional information, or may construct a single trained model M. Alternatively, the input time series data A may include multiple pieces of information selected from tone information, genre information, and difficulty level information as additional information.

[0050] Furthermore, in the above embodiment, the chord estimating device 20 includes the generating unit 23, but the embodiment is not limited to this. A performer can create a score with chords by transcribing the chord sequence information estimated by the estimating unit 22 into a desired score. Therefore, the chord estimating device 20 does not need to include the generating unit 23.

[0051] In the above embodiment, the training data D is trained to estimate chord sequence information when playing on a piano, but the embodiment is not limited to this. The training data D may also be trained to estimate chord sequence information when playing on other instruments such as a guitar or drums.

[0052] In the above embodiment, the user of the chord estimating device 20 is a performer, but the user of the chord estimating device 20 may be, for example, a staff member of a music score production company. Furthermore, machine learning by the training device 10 may be performed in advance by a staff member of the music score production company.

Claims

1. a receiving unit that receives an image of a musical score or time-series data including a sequence of musical notes formed by an operation using an operation unit; an estimation unit that estimates chord sequence information indicating a chord sequence corresponding to the musical note sequence, the chord sequence information including chord change timings, based on the time-series data; A code estimation device comprising:

2. a receiving unit that receives time-series data including a musical note sequence consisting of a plurality of musical notes through an operation using an operation unit; an estimation unit that estimates chord sequence information indicating a chord sequence corresponding to the musical note sequence, the chord sequence information including chord change timings, based on the time-series data; A code estimation device comprising:

3. the time-series data includes genre information that specifies the genre of music expressed by the sequence of notes; The chord estimation device according to claim 1 or 2, wherein the estimation unit estimates the chord string information based on the time-series data including genre information.

4. the time-series data includes key information that specifies the key of the music expressed by the sequence of notes; The chord estimation device according to claim 1 or 2, wherein the estimation unit estimates the chord string information based on the time-series data including key information.

5. the time-series data includes difficulty level information that specifies the difficulty level of the musical score represented by the sequence of notes, The chord estimation device according to claim 1 or 2, wherein the estimation unit estimates the chord string information based on the time-series data including difficulty level information.

6. The chord estimating device according to any one of claims 1 to 5, further comprising a generating unit that generates musical score information indicating a chord-attached musical score to which the chord sequence information is attached so as to correspond to each note of the musical note sequence.

7. Accepting an image of a musical score or time-series data including a sequence of multiple notes via an operation using an operation unit; a computer-executable chord estimation method for estimating, based on the time-series data, information indicating a chord sequence corresponding to the sequence of notes, the chord sequence information including chord change timings recorded therein;

8. Accepting time series data including a sequence of notes consisting of multiple notes through an operation using the operation unit; a computer-executable chord estimation method for estimating, based on the time-series data, information indicating a chord sequence corresponding to the sequence of notes, the chord sequence information including chord change timings recorded therein;

9. the time-series data includes genre information that specifies the genre of music expressed by the sequence of notes; 9. The computer-executable chord estimation method according to claim 7, wherein said estimating comprises estimating said chord string information based on said time-series data including genre information.

10. the time-series data includes key information that specifies the key of the music expressed by the sequence of notes; 9. The computer-implemented chord estimation method according to claim 7, wherein said estimating comprises estimating said chord string information based on said time-series data including key information.

11. the time-series data includes difficulty level information that specifies the difficulty level of the musical score represented by the sequence of notes, 9. The computer-executable chord estimation method according to claim 7, wherein said estimating comprises estimating said chord string information based on said time-series data including difficulty level information.

12. The chord estimating method executed by a computer according to any one of claims 7 to 11, further comprising generating musical score information indicating a chord-attached musical score to which the chord sequence information is attached so as to correspond to each note of the musical note sequence.

Images