Music inference device, music inference method, music inference program, model generation device, model generation method, and model generation program

Abstract

To provide a technique for reducing the probability of occurrence of a temporal error in inference for music.SOLUTION: A music inference apparatus according to an aspect of the present invention executes a step of acquiring target data including an input token sequence arranged so as to indicate at least a part of music, the input token sequence including a plurality of mora tokens arranged so as to indicate positions of morae of the music, a step of generating an output token sequence indicating an inference result for the music from the input token sequence included in the target data by using a trained inference model, and a step of outputting the acquired inference result.SELECTED DRAWING: Figure 6A

Description

[Technical Field]

[0001] The present invention relates to a music piece inference device, a music piece inference method, a music piece inference program, a model generation device, a model generation method, and a model generation program. [Background technology]

[0002] For example, music inference, such as generating arranged music, generating music scores, and estimating music attributes, has traditionally been performed primarily manually by humans. However, performing all music inference tasks manually would result in high costs. Therefore, efforts are underway to develop methods for automating at least part of music inference tasks using computer technology.

[0003] For example, Patent Document 1 proposes a technology for automatically generating accompaniment (backing) data through arrangement. Furthermore, in recent years, AI (artificial intelligence) technology has come to be used in methods for automating inference work for music. For example, Non-Patent Document 1 proposes a method for automatically generating music using a model trained by machine learning. These technologies can reduce the cost of inference work for music. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-58594 [Non-patent literature]

[0005] [Non-Patent Document 1] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck, "Music Transformer", [online], [Retrieved September 24, 2021], Internet<URL: https: / / arxiv.org / abs / 1809.04281> [Non-patent document 2] Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, Karen Simonyan, "This Time with Feeling: Learning Expressive Musical Performance", [online], [Retrieved September 24, 2021], Internet<URL: https: / / arxiv.org / abs / 1808.03715> [Non-patent document 3] Yu-Siang Huang, Yi-Hsuan Yang, "Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions", [online], [Retrieved September 24, 2021], Internet<URL: https: / / arxiv.org / abs / 2002.00212> Summary of the Invention [Problem to be solved by the invention]

[0006] The present inventors have discovered that conventional methods for inferring music using AI technology have the following problems. Specifically, conventional methods using AI technology generally tokenize information representing music, such as musical notes, and input the resulting token sequence into a trained model, which then performs computational processing on the trained model (e.g., Non-Patent Documents 2 and 3). This computational processing results in a token sequence representing the inference result being obtained from the trained model. At this time, a timing error may occur in the obtained inference result. As an example, consider a case in which a trained model is used to automatically generate an accompaniment from a piece of music. In this case, an error may occur in which the playing time of the generated accompaniment does not match the playing time of the original piece of music. When such a timing error occurs, it is difficult to identify the location and cause of the error, making it difficult to correct the obtained inference result (e.g., correcting the duration of the obtained accompaniment data in the above case). Note that timing errors are not limited to cases in which an accompaniment is automatically generated; similar problems may occur in any situation in which a trained model is used to perform inference processing on a piece of music.

[0007] In one aspect, the present invention has been made in view of the above circumstances, and its purpose is to provide a technique for reducing the probability of occurrence of a temporal error in inferring music. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the present invention employs the following configuration.

[0009] That is, a musical piece inference device according to one aspect of the present invention includes: a data acquisition unit that acquires target data including an input token sequence arranged to represent at least a portion of a musical piece, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions of the musical piece; an inference unit that generates an output token sequence representing an inference result for the musical piece from the input token sequence included in the target data by using a trained inference model; and an output unit that outputs the inference result. Note that the output token sequence may also be configured to include metrical tokens.

[0010] In the musical composition estimation device according to the above aspect, each of the plurality of metrical tokens may be arranged at a position of a bar line and a beat of the musical composition in the input token sequence.

[0011] In the musical composition inference device according to the above aspect, the input token sequence may be generated corresponding to at least a portion of a sequence of notes of the musical composition, and the output token sequence may be generated to indicate at least a portion of a sequence of notes of an arranged musical composition as a result of inference on the musical composition.

[0012] In the musical piece inference device according to the above aspect, the input token sequence may be generated corresponding to a sequence of notes of at least a portion of the musical piece, and the output token sequence may be generated to indicate a result of estimating local attributes of at least a portion of the musical piece as a result of inference on the musical piece.

[0013] In the musical composition inference device according to the above aspect, the input token sequence may be generated corresponding to at least a portion of a musical note sequence of the musical composition, and the output token sequence may be generated to represent at least a portion of a musical score of the musical composition as a result of inference on the musical composition.

[0014] In the musical composition inference device according to the above aspect, the input token sequence may be generated corresponding to at least a portion of an element sequence of the musical composition, and the output token sequence may be generated to indicate at least a portion of an arranged musical note sequence of the musical composition as a result of inference on the musical composition.

[0015] The aspects of the present invention are not limited to a music inference device configured to use a trained inference model, and one aspect of the present invention may be a model generation device configured to generate a trained inference model used in any of the above aspects.

[0016] For example, a model generation device according to one aspect of the present invention comprises: a training data acquisition unit that acquires multiple training data sets, each of which is composed of a combination of training data and a correct answer label, wherein the training data includes an input token sequence arranged to indicate at least a portion of a training piece of music, the input token sequence includes a plurality of metrical tokens arranged to indicate the position of the beats of the music, and the correct answer label is configured to indicate the true value of an output token sequence corresponding to the result of inference for the music; and a training processing unit that performs machine learning of an inference model using the multiple acquired training data sets, wherein the machine learning is configured by training the inference model for each of the training data sets so that the output token sequence generated by the inference model from the input token sequence included in the training data conforms to the true value indicated by the correct answer label.

[0017] In the model generating device according to the above aspect, each of the plurality of metrical tokens may be placed at a position corresponding to a bar line and a beat of the music piece in the input token sequence.

[0018] In the model generation device according to the above aspect, the input token sequence included in the training data of each of the learning data sets may be generated corresponding to at least a portion of a sequence of notes in the musical piece, and the output token sequence of the correct answer label of each of the learning data sets may be configured to indicate the true value of at least a portion of a sequence of notes in an arranged musical piece as the true value of the inference result for the musical piece.

[0019] In the model generation device according to the above aspect, the input token sequence included in the training data of each of the learning data sets may be generated corresponding to a sequence of notes of at least a portion of the music piece, and the output token sequence of the correct answer label of each of the learning data sets may be configured to indicate the true value of the result of estimating local attributes of at least a portion of the music piece as the true value of the result of inference for the music piece.

[0020] In the model generation device according to the above aspect, the input token sequence included in the training data of each of the learning data sets may be generated corresponding to at least a portion of a sequence of notes of the music piece, and the output token sequence of the correct answer label of each of the learning data sets may be configured to indicate the true value of at least a portion of the musical score of the music piece as the true value of the result of inference for the music piece.

[0021] In the model generation device according to the above aspect, the input token sequence included in the training data of each of the learning data sets may be generated corresponding to at least a portion of an element sequence of the musical piece, and the output token sequence of the correct answer label of each of the learning data sets may be configured to indicate the true value of at least a portion of a note sequence of the arranged musical piece as the true value of the inference result for the musical piece.

[0022] As another embodiment of the music inference device and model generation device according to each of the above embodiments, one aspect of the present invention may be an information processing method, an information processing system, a program, or a storage medium storing such a program and readable by a computer or other device or machine, etc. Here, a storage medium readable by a computer, etc. is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0023] For example, a musical piece inference method according to one aspect of the present invention is an information processing method that includes the steps of: acquiring target data including an input token sequence arranged to represent at least a portion of a musical piece, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions of the musical piece; generating an output token sequence representing an inference result for the musical piece from the input token sequence included in the target data using a trained inference model; and outputting the inference result.

[0024] Also, for example, a music inference program according to one aspect of the present invention is a program for causing a computer to execute the steps of: acquiring target data including an input token sequence arranged to represent at least a portion of a piece of music, the input token sequence including a plurality of metrical tokens arranged to represent the positions of the beats of the piece of music; generating an output token sequence representing the result of inference for the piece of music from the input token sequence included in the target data by using a trained inference model; and outputting the result of the inference.

[0025] Also, for example, a model generation method according to one aspect of the present invention is an information processing method in which a computer executes the following steps: acquiring multiple training data sets, each consisting of a combination of training data and a correct answer label, wherein the training data includes an input token sequence arranged to indicate at least a portion of a training piece of music, the input token sequence including a plurality of metrical tokens arranged to indicate the position of the beats of the music, and the correct answer label is configured to indicate the true value of an output token sequence corresponding to the result of inference for the music; and performing machine learning of an inference model using the acquired multiple training data sets, wherein the machine learning is configured by training the inference model for each training data set so that the output token sequence generated by the inference model from the input token sequence included in the training data matches the true value indicated by the correct answer label.

[0026] Furthermore, for example, a model generation program according to one aspect of the present invention is a program for executing the following steps on a computer: acquiring a plurality of training data sets, each of which is composed of a combination of training data and correct labels, wherein the training data includes an input token sequence arranged to indicate at least a portion of a piece of music to be learned, the input token sequence including a plurality of metrical tokens arranged to indicate the position of the beats of the music, and the correct labels are configured to indicate the true value of an output token sequence corresponding to the result of inference for the music; and performing machine learning of an inference model using the acquired plurality of training data sets, wherein the machine learning is configured by training the inference model for each of the training data sets so that the output token sequence generated by the inference model from the input token sequence included in the training data conforms to the true value indicated by the correct label. [Effects of the Invention]

[0027] According to the present invention, it is possible to provide a technique for reducing the probability of occurrence of a time error in inference about a piece of music. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 shows a schematic diagram of an example of a situation in which the present invention is applied. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a model generating device according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of a musical piece estimation device according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a software configuration of the model generating device according to the embodiment. [Figure 5] FIG. 5 is a musical score showing an example of a piece of music. [Figure 6A] FIG. 6A shows an example of an input token sequence generated from the music piece of FIG. [Figure 6B] FIG. 6B shows an example of an input token sequence generated from the music piece of FIG. [Figure 7] FIG. 7 is a musical score showing an example of an arranged piece of music (inference result). [Figure 8A] FIG. 8A shows an example of the truth values ​​of the output token sequence corresponding to the music piece of FIG. [Figure 8B] FIG. 8B shows an example of the truth values ​​of the output token sequence corresponding to the music piece of FIG. [Figure 9] FIG. 9 is a schematic diagram showing an example of the configuration of an inference model according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the software configuration of a musical piece deduction device according to an embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an example of a processing procedure of the model generating device according to the embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of a processing procedure of the musical piece estimation device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0029] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in the present embodiment are described in natural language, more specifically, they are specified using pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

[0030] §1 Application Examples 1 is a diagram illustrating an example of a situation in which the present invention is applied. As shown in FIG. 1, an inference system 100 according to this embodiment includes a model generation device 1 and a music piece inference device 2.

[0031] The model generation device 1 according to this embodiment is a computer configured to generate, through machine learning, a trained inference model 5 for performing an inference task on a piece of music. First, the model generation device 1 acquires multiple training datasets 3. Each training dataset 3 is composed of a combination of training data 31 and a ground truth label 32. The training data 31 is configured to include an input token sequence arranged to represent at least a portion of a training piece of music. The input token sequence includes multiple metrical tokens arranged to represent the metrical positions of the piece of music. The ground truth label 32 is configured to indicate the true value of an output token sequence corresponding to the result of inference on the piece of music.

[0032] Next, the model generation device 1 performs machine learning of the inference model 5 using the acquired multiple training datasets 3. The machine learning is performed by training the inference model 5 for each training dataset 3 so that the output token sequence generated by the inference model 5 from the input token sequence included in the training data 31 matches the true value indicated by the corresponding correct answer label 32. This machine learning process makes it possible to generate a trained inference model 5 that has acquired the ability to perform inference tasks for music pieces.

[0033] On the other hand, the musical piece inference device 2 according to this embodiment is a computer configured to perform an inference task for a musical piece using a trained inference model 5. First, the musical piece inference device 2 acquires target data 221 including an input token sequence arranged to represent at least a portion of a musical piece. The input token sequence is configured to include a plurality of metrical tokens arranged to represent the positions of the musical pieces. Next, by using the trained inference model 5, the musical piece inference device 2 generates inference result data including an output token sequence indicating the result of inference for the musical piece from the input token sequence included in the target data 221. The musical piece inference device 2 outputs the acquired inference result.

[0034] The input token sequences of the training data 31 and the target data 221 may be acquired as appropriate depending on the embodiment. As an example, a piece of music may be acquired as performance information expressed in other ways, such as encoded data (e.g., MIDI) or sheet music. The input token sequences of the training data 31 and the target data 221 may be generated from the acquired performance information by a conversion process such as natural language processing. The conversion process may be executed by a computer other than each device (1, 2). Furthermore, the conversion process may be executed at any timing. Each device (1, 2) may acquire the input token sequence directly, or may acquire performance information expressed in other ways and generate the input token sequence from the acquired performance information.

[0035] The inference task to be performed by the inference model 5 may include any inference for at least a part of a piece of music. The input token sequence and the output token sequence may be configured appropriately depending on the inference task.

[0036] As an example, the inference task may be generating a note sequence of an arranged piece of music from a note sequence of a piece of music. The note sequence is a sequence of notes that constitutes the piece of music. The arrangement may be, for example, a change in the difficulty level of the piece of music, a reduction (such as converting a multi-instrument note sequence to a solo instrument note sequence), or the like. In this case, the input token sequences included in the training data 31 and the target data 221 may be configured to correspond to at least a portion of the note sequence of the piece of music. The output token sequence in the inference result data may be generated as a result of inference on the piece of music, so as to indicate at least a portion of the note sequence of the arranged piece of music. The output token sequence of the correct answer label 32 may be configured to indicate the true value of at least a portion of the note sequence of the arranged piece of music, corresponding to the associated training data 31, as the true value of the inference result.

[0037] As another example, the inference task may be to estimate local attributes of a piece of music from a sequence of notes in the piece of music. The local attributes may be, for example, chords, keys, beats, or the timing of changes therein. In this case, the input token sequences included in the training data 31 and the target data 221 may be configured to correspond to at least a portion of a sequence of notes in the piece of music. The output token sequence in the inference result data may be generated as a result of inference on the piece of music, so as to indicate the result of estimating local attributes of at least a portion of the piece of music. The output token sequence of the correct answer label 32 may be configured to indicate the true value of the result of inference, corresponding to the associated training data 31, of the result of the inference.

[0038] As another example, the inference task may be generating a musical score from a sequence of musical notes in a piece of music. In this case, the input token sequences included in the training data 31 and the target data 221 may be configured to correspond to at least a portion of the musical note sequence in the piece of music. The output token sequence in the inference result data may be generated to indicate at least a portion of the musical score in the piece of music as a result of inference on the piece of music. The output token sequence of the correct answer label 32 may be configured to indicate the true value of at least a portion of the musical score in the piece of music, corresponding to the associated training data 31, as the true value of the inference result.

[0039] As another example, the inference task may be generating a sequence of notes of an arranged piece of music from a sequence of musical elements. The sequence of elements is a sequence of musical elements (materials) that make up the piece of music. The elements may be, for example, a melody, a chord, or a rhythm. In this case, the input token sequences included in the training data 31 and the target data 221 may be configured to correspond to at least a portion of the sequence of elements of the piece of music. The output token sequence in the inference result data may be generated as a result of inference on the piece of music, so as to indicate at least a portion of the sequence of notes of the arranged piece of music. The output token sequence of the correct answer label 32 may be configured to indicate, as the true value of the inference result, the true value of at least a portion of the sequence of notes of the arranged piece of music, corresponding to the associated training data 31.

[0040] As another example, the inference task may be to generate a sequence of notes from a sequence of musical elements that represents a motif. The generated sequence of notes may be configured to represent a melody or an arranged musical piece. In this case, the input token sequences included in the training data 31 and the target data 221 may be configured to correspond to at least a portion of the sequence of elements of the musical piece. The output token sequence in the inference result data may be generated to represent at least a portion of the musical note sequence of the musical piece as a result of inference on the musical piece. The output token sequence of the correct answer label 32 may be configured to represent the true value of at least a portion of the musical note sequence of the musical piece, corresponding to the associated training data 31, as the true value of the inference result.

[0041] Each of the multiple metrical tokens is appropriately positioned in the input token sequence to indicate the metrical structure of the music. Specifically, each metrical token is positioned in the input token sequence to indicate at least one of the positions of a bar line and a beat of the music. A bar line indicates the division of a measure. A measure is a division of a musical score of an appropriate length to make it easier to read. A beat is a unit that divides the temporal continuity of music. In one example, each metrical token may be positioned to indicate only one of a bar line and a beat. This allows the metrical structure of the music to be understood using each metrical token as a clue. However, metrical structures vary depending on the music. Some music pieces have rhythmic changes midway through. It is difficult to fully understand the metrical structure of various types of music using only one of a bar line and a beat. Therefore, preferably, each metrical token is positioned at the position of a bar line and a beat in the input token sequences of the training data 31 and the target data 221.

[0042] The tokens in the input token string and the output token string may be composed of symbols such as numbers, letters, figures, etc. Similarly, each metrical token may be composed of symbols such as numbers, letters, figures, etc. As long as they are recognizable by a computer, the symbols and data formats used for the tokens are not particularly limited and may be selected appropriately depending on the embodiment. Figures 6A, 6B, 8A, and 8B, which will be described later, show examples of each token.

[0043] In conventional methods, the token sequence input to the trained model does not contain information indicating the metrical structure of the music. Therefore, while it was possible to perform inference with a certain degree of accuracy for music with a predefined metrical structure, it was difficult to perform appropriate inference for music with various types of metrical structures, such as music with changing beats or music with a metrical structure different from the training data. This was presumed to be one of the main causes of the above-mentioned temporal error.

[0044] In contrast, in this embodiment, the input token sequence used for inference is configured to include multiple metrical tokens that indicate the position of the beats in a piece of music, as described above. This allows the inference model 5 to identify the metrical structure of a piece of music and then perform inference processing on the piece of music. As a result, the model generation device 1 can generate a trained inference model 5 that is less likely to produce temporal errors due to the metrical structure. The music inference device 2 uses this trained inference model 5 to perform an inference task on target data 221 that includes multiple metrical tokens. This reduces the probability of temporal errors occurring in inference tasks for music.

[0045] In one example, the model generation device 1 can generate a trained inference model 5 that has acquired the ability to perform inference processes such as generating a note sequence of an arranged piece of music from the note sequence of the piece of music to be inferred, estimating local attributes of the piece of music from the note sequence of the piece of music to be inferred, generating musical scores from the note sequence of the piece of music to be inferred, and generating a note sequence of an arranged piece of music from the element sequence of the piece of music to be inferred, and is less likely to have temporal errors due to metrical structure. The music inference device 2 can reduce the probability of temporal errors occurring when performing these inference processes.

[0046] 1, the model generation device 1 and the musical piece deduction device 2 are connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of exchanging data between the model generation device 1 and the musical piece deduction device 2 is not limited to this example and may be appropriately selected depending on the embodiment. For example, data may be exchanged between the model generation device 1 and the musical piece deduction device 2 using a storage medium.

[0047] 1, the model generation device 1 and the musical piece inference device 2 are each configured as separate computers. However, the configuration of the inference system 100 according to this embodiment is not limited to this example and may be determined appropriately depending on the embodiment. For example, the model generation device 1 and the musical piece inference device 2 may be integrated into one computer. Furthermore, for example, at least one of the model generation device 1 and the musical piece inference device 2 may be configured as multiple computers. When configured as multiple computers, the distribution of information processing may be determined appropriately depending on the embodiment.

[0048] §2 Configuration example [Hardware configuration] <Model generation device> Fig. 2 schematically illustrates an example of the hardware configuration of the model generation device 1 according to this embodiment. As shown in Fig. 2, the model generation device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an external interface 14, an input device 15, an output device 16, and a drive 17 are electrically connected. In Fig. 2, the communication interface and the external interface are referred to as a "communication I / F" and an "external I / F."

[0049] The control unit 11 includes a CPU (Central Processing Unit), which is an example of a hardware processor (processor resource), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. The storage unit 12 is an example of memory, and is configured, for example, with a hard disk drive, a solid state drive, etc. In this embodiment, the storage unit 12 stores various information such as a model generation program 81, multiple training datasets 3, training result data 125, etc.

[0050] The model generation program 81 is a program for causing the model generation device 1 to execute machine learning information processing (FIG. 11) described below for generating a trained inference model 5. The model generation program 81 includes a series of instructions for the information processing. Multiple training datasets 3 are used to generate the trained inference model 5. The learning result data 125 indicates information about the generated trained inference model 5. In this embodiment, the learning result data 125 is generated as a result of executing the model generation program 81. Details will be described later.

[0051] The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The model generation device 1 can use the communication interface 13 to perform data communication via a network with other information processing devices. The external interface 14 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 14 may be selected arbitrarily.

[0052] The model generation device 1 may be connected to a device for obtaining each learning data set 3 via at least one of the communication interface 13 and the external interface 14. As an example, the input token sequence of the training data 31 may be generated from performance information obtained by an electronic musical instrument. When the generation of the input token sequence from this performance information is executed by the model generation device 1, the model generation device 1 may be connected to the electronic musical instrument via at least one of the communication interface 13 and the external interface 14, and the performance information for generating the training data 31 may be collected by the electronic musical instrument.

[0053] The input device 15 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 16 is a device for outputting, for example, a display, a speaker, etc. An operator such as a user can operate the model generation device 1 by using the input device 15 and the output device 16.

[0054] The drive 17 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading various information, such as programs, stored in a storage medium 91. The storage medium 91 is a medium that stores information, such as programs, electrically, magnetically, optically, mechanically, or chemically, so that a computer or other device, machine, or the like can read the stored information. At least one of the model generation program 81 and the multiple training datasets 3 may be stored in the storage medium 91. The model generation device 1 may acquire at least one of the model generation program 81 and the multiple training datasets 3 from the storage medium 91. Note that FIG. 2 illustrates a disk-type storage medium, such as a CD or a DVD, as an example of the storage medium 91. However, the type of the storage medium 91 is not limited to a disk-type storage medium and may be other than a disk-type storage medium. Examples of storage media other than a disk-type storage medium include semiconductor memories, such as flash memories. The type of the drive 17 may be selected arbitrarily depending on the type of the storage medium 91.

[0055] Note that, with regard to the specific hardware configuration of the model generation device 1, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, an FPGA (field-programmable gate array), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the communication interface 13, the external interface 14, the input device 15, the output device 16, and the drive 17 may be omitted. The model generation device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the model generation device 1 may be an information processing device designed specifically for the service to be provided, as well as a general-purpose server device, a PC (Personal Computer), or the like.

[0056] <Music inference device> 3 is a schematic diagram illustrating an example of the hardware configuration of the musical piece estimation device 2 according to this embodiment. As shown in Fig. 3, the musical piece estimation device 2 according to this embodiment is a computer to which a control unit 21, a storage unit 22, a communication interface 23, an external interface 24, an input device 25, an output device 26, and a drive 27 are electrically connected.

[0057] The control unit 21 to the drive 27 and the storage medium 92 of the musical piece inference device 2 may be configured similarly to the control unit 11 to the drive 17 and the storage medium 91 of the model generation device 1, respectively. The control unit 21 includes a CPU, which is an example of a hardware processor, RAM, ROM, etc., and is configured to execute various information processes based on programs and data. The storage unit 22 is configured, for example, with a hard disk drive, solid state drive, etc. In this embodiment, the storage unit 22 stores various information such as a musical piece inference program 82, learning result data 125, etc.

[0058] The music inference program 82 is a program for causing the music inference device 2 to execute information processing ( FIG. 12 ), which will be described later, for performing an inference task for a music piece using a trained inference model 5. The music inference program 82 includes a series of instructions for the information processing. At least one of the music inference program 82 and the learning result data 125 may be stored in the storage medium 92. Furthermore, the music inference device 2 may acquire at least one of the music inference program 82 and the learning result data 125 from the storage medium 92.

[0059] The musical composition estimation apparatus 2 may be connected to a device for obtaining the target data 221 via at least one of a communication interface 23 and an external interface 24. As an example, an input token sequence of the target data 221 may be generated from performance information obtained by an electronic musical instrument. When the generation of the input token sequence from this performance information is executed in the musical composition estimation apparatus 2, the musical composition estimation apparatus 2 may be connected to the electronic musical instrument via at least one of the communication interface 23 and the external interface 24. Furthermore, the musical composition estimation apparatus 2 may receive operations and inputs from an operator such as a user by using an input device 25 and an output device 26.

[0060] Note that with regard to the specific hardware configuration of the musical piece estimation device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, or the like. The storage unit 22 may be configured with RAM and ROM included in the control unit 21. At least one of the communication interface 23, external interface 24, input device 25, output device 26, and drive 27 may be omitted. The musical piece estimation device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. Furthermore, the musical piece estimation device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, etc.

[0061] [Software configuration] <Model generation device> 4 schematically illustrates an example of the software configuration of the model generation device 1 according to this embodiment. The control unit 11 of the model generation device 1 interprets instructions included in the model generation program 81 stored in the storage unit 12 and executes control processing according to the interpreted instructions. As a result, the model generation device 1 according to this embodiment is configured to include a learning data acquisition unit 111, a learning processing unit 112, and a storage processing unit 113 as software modules. That is, in this embodiment, the software modules of the model generation device 1 are realized by the control unit 11 (CPU).

[0062] The training data acquisition unit 111 is configured to acquire multiple training data sets 3. Each training data set 3 is composed of a combination of training data 31 and a correct answer label 32. The training data 31 includes an input token sequence arranged to represent at least a portion of a training piece of music. The input token sequence includes multiple metrical tokens arranged to represent the metrical positions of the music. The at least a portion of the music may be defined as having a predetermined length, such as four bars. The correct answer label 32 is configured to represent the true value of an output token sequence corresponding to the result of inference on the music.

[0063] The learning processing unit 112 is configured to perform machine learning of the inference model 5 using the acquired multiple learning datasets 3. The machine learning is configured by training the inference model 5 for each learning dataset 3 so that an output token sequence generated by the inference model 5 from an input token sequence included in the training data 31 matches the true value indicated by the corresponding correct label 32. Upon completion of this machine learning process, a trained inference model 5 that has acquired the ability to perform a desired inference task is generated.

[0064] The storage processing unit 113 is configured to generate information about the trained inference model 5 generated by machine learning as learning result data 125, and to store the generated learning result data 125 in a predetermined storage area. The learning result data 125 may be appropriately configured to include information for reproducing the trained inference model 5.

[0065] (An example of a token) The tokens constituting the input token string and the output token string may be any symbols, such as numbers, letters, figures, etc. The symbols (token representations) and data formats used for the tokens are not particularly limited as long as they are recognizable by a computer, and may be selected appropriately depending on the embodiment. The same applies to metrical tokens. Two tokenization methods, namely, movement-based and note-based, are exemplified below as examples of tokenization methods.

[0066] FIG. 5 is a musical score showing an example of at least a portion of a piece of music. FIG. 6A shows an example of an input token sequence generated from the music of FIG. 5 using a motion-based tokenization scheme. FIG. 6B shows an example of an input token sequence generated from the music of FIG. 5 using a note-based tokenization scheme. FIG. 7 is a musical score showing an example of an arranged piece of music (inference result) obtained from the music of FIG. 5 as an example of an inference task. FIG. 8A shows an example of the truth value of an output token sequence obtained corresponding to the music of FIG. 7 using a motion-based tokenization scheme. FIG. 8B shows an example of the truth value of an output token sequence obtained corresponding to the music of FIG. 7 using a note-based tokenization scheme.

[0067] The action-based tokenization method is a method of tokenizing so as to indicate actions corresponding to musical notes or elements of a piece of music. Table 1 shows an example of the types of tokens and their representations in the action-based tokenization method. On the other hand, the note-based tokenization method is a method of tokenizing so as to directly indicate musical notes in a piece of music. Table 2 shows an example of the types of tokens and their representations in the note-based tokenization method. Note that the following token types and representations are merely examples and may be changed as appropriate depending on the embodiment. [Table 1] [Table 2]

[0068] Either of the above two methods may be employed for the tokenization method and token representation of the input token sequence and the output token sequence. As an example of a method for acquiring each training dataset 3, music data representing at least a portion of the music shown in FIG. 5 may be acquired as appropriate. The format of the music data may be selected as appropriate depending on the embodiment. For example, the music data may be obtained in the form of encoded data (e.g., MIDI), sheet music, or the like. Training data 31 for each training dataset 3 may be generated as appropriate from the acquired music data to include the input token sequence shown in FIG. 6A or 6B. Furthermore, correct answer data indicating the true value of the inference result for the music shown in FIG. 7 (in the example of FIG. 7, the arrangement result of the music) may be obtained corresponding to at least a portion of the music shown in FIG. 5. Correct answer labels 32 for each training dataset 3 may be generated as appropriate from the acquired correct answer data to include the true value of the output token sequence shown in FIG. 8A or 8B. The true values ​​of the input token sequence and the output token sequence may be generated using any conversion process, such as natural language processing. The truth values ​​of the input and output token sequences may be generated manually by a human being.

[0069] 7, 8A, and 8B show an example of a situation in which generating an arranged piece of music is adopted as the inference task. However, as mentioned above, the inference task to be performed by the inference model 5 does not have to be limited to this. Similarly, for other inference tasks, the true values ​​of the inference results and output token sequences may be obtained as appropriate.

[0070] Among the tokens included in the input token strings and output token strings shown in FIGS. 6A, 6B, 8A, and 8B, "bar" and "beat" are examples of metrical tokens. "Bar" is an example of a token indicating a bar line, and "beat" is an example of a token indicating a beat (meter). As illustrated in FIGS. 6A and 6B, the input token string is configured to include multiple metrical tokens. This makes it possible to identify the metrical structure of a piece of music. Furthermore, as illustrated in FIGS. 8A and 8B, the output token string may also be configured to include metrical tokens. Note that these representations of metrical tokens are merely examples. The representation of metrical tokens is not limited to these examples and may be determined appropriately depending on the embodiment.

[0071] The input token sequence and the output token sequence may use the same tokenization scheme and the same token representation. In the above example, an action-based or note-based tokenization scheme may be adopted for both the input token sequence and the output token sequence. However, the format of the input token sequence and the output token sequence is not limited to this example. The input token sequence and the output token sequence do not necessarily have to use the same tokenization scheme and the same token representation. Different tokenization schemes and different token representations may be adopted for the input token sequence and the output token sequence.

[0072] As long as a computer can recognize at least a portion of the music piece to be inferred, the format of the tokens used in the input token sequence need not be particularly limited and may be determined appropriately depending on the embodiment. As long as a computer can recognize the inference result, the format of the tokens used in the output token sequence need not be particularly limited and may be determined appropriately depending on the embodiment. Furthermore, as long as a computer can recognize the metrical structure, the format of the metrical tokens need not be particularly limited and may be determined appropriately depending on the embodiment.

[0073] (An example of an inference model) FIG. 9 schematically illustrates an example of the configuration of the inference model 5 according to this embodiment. The inference model 5 is configured by a machine learning model having parameters adjusted by machine learning. The type of machine learning model is not particularly limited and may be selected appropriately depending on the embodiment. The structure of the machine learning model is not particularly limited and may be determined appropriately depending on the embodiment as long as it is configured to accept an input of an input token sequence and output an output token sequence indicating an inference result. As an example, as shown in FIG. 9, the inference model 5 may have a configuration based on the Transformer proposed in the reference document "Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017." The Transformer is a machine learning model that processes sequence data (such as natural language) and has an attention-based configuration.

[0074] In the example of FIG. 9 , the inference model 5 includes an encoder 50 and a decoder 55. The encoder 50 has a structure configured by stacking multiple blocks, each of which has a multi-head attention layer for self-attention and a feed-forward layer. Meanwhile, the decoder 55 has a structure configured by stacking multiple blocks, each of which has a masked multi-head attention layer for self-attention, a multi-head attention layer for source-target attention, and a feed-forward layer. As shown in FIG. 9 , each layer of the encoder 50 and the decoder 55 may be provided with an addition and normalization layer. Each layer may include one or more nodes, and a threshold may be set for each node. The threshold may be expressed by an activation function. Furthermore, weights (connection weights) may be set for connections between nodes in adjacent layers. The weights and thresholds for connections between nodes are examples of parameters of the inference model 5.

[0075] In the example shown in Figure 9, the inference model 5 is configured to accept tokens contained in an input token sequence in order from the beginning. The tokens input to the inference model 5 are each converted into a vector having a predetermined number of dimensions by an input embedding process, and a value specifying their position within the music (within a phrase) is assigned by a position encoding process, after which the tokens are input to the encoder 50. The encoder 50 repeatedly performs processing on the input using a multi-head attention layer and a feedforward layer for the number of blocks to obtain feature representations, and supplies the obtained feature representations to the next-stage decoder 55 (multi-head attention layer).

[0076] The decoder 55 (masked multi-head attention layer) is supplied with the input from the encoder 50 as well as known (past) outputs from the decoder 55. That is, the inference model 5 illustrated in FIG. 9 is configured to have a recursive structure. The decoder 55 repeatedly performs processing on the input using the masked multi-head attention layer, the multi-head attention layer, and the feedforward layer for the number of blocks to obtain and output feature representations. The output from the decoder 55 is transformed in the linear layer and softmax layer and obtained as tokens indicating the inference results.

[0077] The learning processing unit 112 is configured to perform machine learning of the inference model 5 for each learning data set 3, using an input token sequence (plurality of tokens) included in the training data 31 as input data and a true value of an output token sequence indicated by a corresponding correct label 32 as a teacher signal. Specifically, the learning processing unit 112 is configured to input the input token sequence included in the training data 31 to the inference model 5 for each learning data set 3, and train the inference model 5 so that an output token sequence obtained by executing a calculation process of the inference model 5 matches the true value indicated by the corresponding correct label 32. In other words, the learning processing unit 112 is configured to adjust parameter values ​​of the inference model 5 for each learning data set 3 so that an error between an output token sequence generated by the inference model 5 from the input token sequence included in the training data 31 and the true value indicated by the corresponding correct label 32 is reduced. Any method, such as backpropagation, may be used to adjust the parameters. In addition, multiple normalization techniques (e.g., label smoothing, residual dropout, attention dropout) may be applied to the machine learning process of the inference model 5.

[0078] <Music inference device> 10 schematically illustrates an example of the software configuration of the musical piece estimation device 2 according to this embodiment. The control unit 21 of the musical piece estimation device 2 interprets instructions included in the musical piece estimation program 82 stored in the storage unit 22, and executes control processing according to the interpreted instructions. As a result, the musical piece estimation device 2 according to this embodiment is configured to include a data acquisition unit 211, an estimation unit 212, and an output unit 213 as software modules. That is, in this embodiment, like the model generation device 1, the software modules of the musical piece estimation device 2 are realized by the control unit 21 (CPU).

[0079] The data acquisition unit 211 is configured to acquire target data 221 including an input token sequence arranged to represent at least a portion of a piece of music. The input token sequence is configured to include a plurality of metrical tokens arranged to represent metrical positions in the piece of music. The input token sequence included in the target data 221 may be generated in a format similar to the input token sequence included in the training data 31 illustrated in FIGS. 6A and 6B.

[0080] The inference unit 212 stores the learning result data 125 and is thus equipped with a trained inference model 5. The inference unit 212 is configured to generate an output token sequence indicating the result of inference on a piece of music from an input token sequence included in the target data 221 by using the trained inference model 5. In the example of FIG. 9 , the inference unit 212 sequentially inputs the input token sequence included in the target data 221 to an encoder 50 of the trained inference model 5 (specifically, after passing through an input embedding layer, to a multi-head attention layer arranged first), and executes arithmetic processing by the encoder 50 and the decoder 55. As a result of this arithmetic processing, the inference unit 212 sequentially acquires tokens output from the trained inference model 5 (in the example of FIG. 9 , the softmax layer arranged last), thereby generating an output token sequence indicating the inference result. During this processing, the output token sequence may be generated using a search method such as beam search, for example. More specifically, the inference unit 212 may generate an output token sequence by retaining n candidate tokens in descending order of score from the probability distribution of values ​​output from the inference model 5 and selecting a candidate token that has the highest integrated score among m consecutive tokens (n ​​and m are integers equal to or greater than 2). The output token sequence generated by the inference unit 212 may be configured in the same format as the output token sequence of the correct label 32 exemplified in Figures 8A and 8B.

[0081] The output unit 213 is configured to output the inference result obtained by the processing of the inference unit 212. The output format of the inference result is not particularly limited and may be determined appropriately depending on the embodiment. As an example, the output token sequence may be output as is. As another example, the output unit 213 may convert the output token sequence into an appropriate format. For example, if the inference task is to generate an arranged piece of music, the output token sequence may be converted into information indicating the music in the form of a sequence of notes, a musical score, or the like of the arranged piece of music. Then, the output unit 213 may output the information obtained by the conversion as the inference result.

[0082] <Other> Each software module of the model generation device 1 and the musical piece inference device 2 according to this embodiment will be described in detail in an operation example below. Note that this embodiment describes an example in which each software module of the model generation device 1 and the musical piece inference device 2 is implemented by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors (e.g., application-specific integrated circuits (ASICs)). Each of the above modules may also be implemented as a hardware module. Furthermore, with regard to the software configuration of the model generation device 1 and the musical piece inference device 2, software modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0083] §3 Example of operation <Model generation device> 11 is a flowchart showing an example of the processing procedure of the model generation device 1 according to this embodiment. The processing procedure of the model generation device 1 described below is an example of a model generation method. However, the processing procedure of the model generation device 1 described below is merely an example, and each step may be changed as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0084] (Step S101) In step S101, the control unit 11 operates as the learning data acquisition unit 111 and acquires a plurality of learning data sets 3.

[0085] Each learning dataset 3 may be generated as needed. Music data representing a piece of music may be obtained in other formats, such as coded data or sheet music, and the input token sequence constituting the training data 31 may be generated as needed from the obtained music data. The correct answer label 32 may be generated as needed to indicate an output token sequence that is the true value of the inference result for the piece of music.

[0086] The process of generating the training datasets 3 may be executed by any computer. In one example, the process of generating each training dataset 3 may be executed by the model generation device 1 (control unit 11). In another example, at least some of the multiple training datasets 3 may be generated by another computer. In this case, the model generation device 1 (control unit 11) may acquire the training datasets 3 generated by the other computer via a network, a storage medium 91, or the like. The number of training datasets 3 to be acquired may be determined appropriately so as to be sufficient for machine learning. After acquiring the multiple training datasets 3, the control unit 11 proceeds to the next step S102.

[0087] (Step S102) In step S102, the control unit 11 operates as the learning processing unit 112 and performs machine learning of the inference model 5 using the acquired multiple learning datasets 3.

[0088] As an example of a specific machine learning process, the control unit 11 inputs an input token sequence included in the training data 31 for each learning data set 3 into the inference model 5 in order from the beginning, and repeatedly executes the calculation process of the inference model 5 to sequentially generate tokens that constitute an output token sequence. Through this calculation process, the control unit 11 can obtain an output token sequence that indicates an inference result corresponding to the training data 31 of each learning data set 3. Next, the control unit 11 calculates the error between the obtained output token sequence and the true value indicated by the corresponding correct label 32, and further calculates the gradient of the calculated error. The control unit 11 calculates the error of the parameter value of the inference model 5 by backpropagating the calculated error gradient using an error backpropagation method. The control unit 11 adjusts the parameter value of the inference model 5 based on the calculated error. The control unit 11 may repeat the adjustment of the parameter value of the inference model 5 using the above series of processes until a predetermined condition is met (for example, a predetermined number of executions, or the sum of the calculated errors is equal to or less than a threshold).

[0089] Through this machine learning, the inference model 5 is trained for each learning dataset 3 so that the output token sequence generated from the input token sequence included in the training data 31 matches the true value indicated by the corresponding correct answer label 32. Therefore, as a result of the machine learning, it is possible to generate a trained inference model 5 that has acquired the ability to perform an inference task so as to match the true value given by the correct answer label 32. When the machine learning process is complete, the control unit 11 proceeds to the next step S103.

[0090] (Step S103) In step S103, the control unit 11 operates as a storage processing unit 113 and generates information about the trained inference model 5 generated by machine learning as learning result data 125. The learning result data 125 holds information for reproducing the trained inference model 5. As an example, the learning result data 125 may include information indicating the values ​​of each parameter of the inference model 5 obtained by adjusting the machine learning. In some cases, the learning result data 125 may include information indicating the structure of the inference model 5. The structure may be specified by, for example, the number of layers, the type of each layer, the number of nodes included in each layer, the connection relationships between nodes in adjacent layers, etc. The control unit 11 stores the generated learning result data 125 in a predetermined storage area.

[0091] The predetermined storage area may be, for example, RAM in the control unit 11, the storage unit 12, an external storage device, a storage medium, or a combination of these. The storage medium may be, for example, a CD, a DVD, or the like, and the control unit 11 may store the learning result data 125 in the storage medium via the drive 17. The external storage device may be, for example, a data server such as a NAS. In this case, the control unit 11 may use the communication interface 13 to store the learning result data 125 in the data server via a network. The external storage device may also be, for example, an external storage device connected to the model generation device 1.

[0092] When the storage of the learning result data 125 is completed, the control unit 11 ends the processing procedure of the model generating device 1 according to this operation example.

[0093] The generated learning result data 125 may be provided to the musical piece inference device 2 at any timing. For example, the control unit 11 may transfer the learning result data 125 to the musical piece inference device 2 as part of the processing of step S103 or separately from the processing of step S103. The musical piece inference device 2 may acquire the learning result data 125 by receiving this transfer. Also, for example, the musical piece inference device 2 may acquire the learning result data 125 by accessing the model generation device 1 or a data server via a network using the communication interface 23. Also, for example, the musical piece inference device 2 may acquire the learning result data 125 via the storage medium 92. Also, for example, the learning result data 125 may be pre-installed in the musical piece inference device 2.

[0094] Furthermore, the control unit 11 may update or generate new learning result data 125 by periodically or irregularly repeating the processes of steps S101 to S103. During this repetition, at least a portion of the multiple learning datasets 3 used for machine learning may be changed, modified, added, deleted, etc. as appropriate. In this way, the control unit 11 may update or generate new trained inference models 5. Then, the control unit 11 may update the learning result data 125 held by the music estimation device 2 by providing the updated or newly generated learning result data 125 to the music estimation device 2 by any method.

[0095] <Music inference device> 12 is a flowchart showing an example of the processing procedure of the musical piece estimation device 2 according to this embodiment. The processing procedure of the musical piece estimation device 2 described below is an example of a musical piece estimation method. However, the processing procedure of the musical piece estimation device 2 described below is merely an example, and each step may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added to the following processing procedure as appropriate depending on the embodiment.

[0096] (Step S201) In step S201, the control unit 21 operates as the data acquisition unit 211 and acquires target data 221 including an input token sequence arranged so as to represent at least a part of a piece of music.

[0097] The input token sequence included in the target data 221 may be generated in any manner. In one example, the input token sequence may be generated from other types of data, such as coded data, musical scores, etc. In another example, the input token sequence may be generated directly in any manner (e.g., manually).

[0098] Furthermore, the target data 221 may be acquired via any route. In one example, the input token sequence may be generated in the musical composition estimation device 2. In this case, the control unit 21 may acquire the target data 221 as a result of executing the generation process. In another example, the generation of the input token sequence may be executed by a computer other than the musical composition estimation device 2. In this case, the control unit 21 may acquire the target data 221, for example, via a network, a storage medium 92, or the like. Upon acquiring the target data 221, the control unit 21 proceeds to the next step S202.

[0099] (Step S202) In step S202, the control unit 21 operates as the inference unit 212 and sets the trained inference model 5 by machine learning with reference to the learning result data 125. The control unit 21 uses the trained inference model 5 to generate an output token sequence indicating the result of inference on the music piece from the input token sequence included in the target data 221. Specifically, the control unit 21 inputs the input token sequence included in the acquired target data 221 to the trained inference model 5 and executes arithmetic processing of the trained inference model 5. In the example of FIG. 9 above, the control unit 21 inputs tokens included in the input token sequence from the beginning to the trained inference model 5 and repeatedly executes forward propagation arithmetic processing of the trained inference model 5, thereby sequentially generating tokens that constitute the output token sequence. As a result of this arithmetic processing, the control unit 21 acquires, from the trained inference model 5, an output token sequence indicating the result of executing an inference task on at least a part of the music piece indicated by the target data 221. When execution of the inference task (arithmetic processing of the trained inference model 5) is completed, the control unit 21 proceeds to the next step S203.

[0100] (Step S203) In step S203, the control unit 21 operates as the output unit 213 and outputs the inference result obtained by the processing of step S202. The output destination and output format are not particularly limited and may be selected appropriately depending on the embodiment. In one example, the output destination may be, for example, RAM, the memory unit 22, a storage medium, an external storage device, another computer, another device, etc. In another example, the control unit 21 may output the output token sequence as is. In another example, the control unit 21 may convert the output token sequence into an appropriate format and output the information obtained by the conversion. As a specific example, if the inference task is to generate an arranged piece of music, the control unit 21 may generate information in the format of a sequence of notes, a musical score, or the like of the arranged piece of music from the obtained output token sequence and output the generated information. If the inference result is obtained in the format of a musical score, the control unit 21 may, for example, output a command to a printing device (not shown) to print the musical score data on a paper medium.

[0101] When the output of the inference result is completed, the control unit 21 ends the processing procedure of the musical piece inference device 2 according to this operation example. The control unit 21 may periodically or irregularly repeat the processing of steps S201 to S203, for example, in response to a request from an operator. During this repetition, at least a portion of the target data 221 (input token sequence) obtained in step S201 may be changed, modified, added, deleted, etc., as appropriate. This allows the control unit 21 to generate an inference result for a new musical piece using the trained inference model 5.

[0102] <Features> As described above, in this embodiment, the input token sequence, which is the training data 31 of each learning dataset 3 used for the machine learning in step S102, is configured to include multiple metrical tokens that indicate the positions of the beats in a piece of music. This allows the inference model 5 to be trained to grasp the metrical structure of a piece of music from the metrical tokens and then perform inference processing on the piece of music. As a result, the machine learning processing in step S102 can generate a trained inference model 5 that is less likely to produce temporal errors due to the metrical structure.

[0103] In step S201, an input token sequence including multiple metrical tokens is acquired as target data 221. Then, in step S202, the input token sequence including multiple metrical tokens is used to infer a piece of music by the trained inference model 5. This allows the trained inference model 5 to grasp the metrical structure of the piece of music to be inferred and then perform inference processing on the piece of music. As a result, the probability of a time error occurring in the inference task for a piece of music in step S202 can be reduced.

[0104] Furthermore, in this embodiment, each metrical token may be arranged to indicate the position of each bar line and beat in the input token sequence. That is, multiple metrical tokens may be arranged so that both bar lines and beats can be recognized. This allows the inference model 5 to completely identify the metrical structure of the music represented by the input token sequence based on each metrical token. Therefore, in the process of step S102, a trained inference model 5 that is less likely to cause temporal errors can be generated. This further reduces the probability of temporal errors occurring in the inference task for music in step S202.

[0105] In this embodiment, the output token sequence may also be configured to include metrical tokens. This allows the location of the temporal error to be easily identified based on the position of the metrical token included in the output token sequence, even if a temporal error occurs in the inference process of step S202. As a result, the obtained inference result can be easily corrected.

[0106] §4 Variations Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect, and it goes without saying that various improvements and modifications can be made without departing from the scope of the present invention.

[0107] For example, in the above embodiment, a machine learning model having a recursive structure configured using a Transformer (FIG. 9) is exemplified as the inference model 5. However, the recursive structure need not be limited to the example shown in FIG. 9. The recursive structure refers to a structure configured to be able to refer to inputs past the target and execute processing on the target (current) input. As long as such calculations are possible, the recursive structure need not be particularly limited and may be determined appropriately depending on the embodiment. In another example, the recursive structure may be configured using a known structure such as an RNN (Recurrent Neural Network) or an LSTM (Long Short-Term Memory).

[0108] Furthermore, in the above embodiment, the inference model 5 is configured to have a recursive structure. However, the configuration of the inference model 5 need not be limited to this example. The recursive structure may be omitted. The inference model 5 may be configured, for example, by a neural network having a known structure such as a fully connected neural network or a convolutional neural network. Furthermore, the manner in which the input token sequence is input to the inference model 5 need not be limited to the example of the above embodiment. In another example, the inference model 5 may be configured to accept multiple tokens included in the input token sequence at once.

[0109] Furthermore, in the above embodiment, as long as an output token sequence indicating an inference result can be generated from an input token sequence corresponding to a piece of music, the type of machine learning model constituting the inference model 5 is not particularly limited and may be selected appropriately depending on the embodiment. Furthermore, in the above embodiment, when the inference model 5 is constituted by a machine learning model having multiple layers, the type of each layer may be selected appropriately depending on the embodiment. For example, each layer may be a convolutional layer, a pooling layer, a dropout layer, a normalization layer, a fully connected layer, etc. Regarding the structure of the inference model 5, components may be omitted, replaced, or added as appropriate.

[0110] In the above embodiment, the inference model 5 may be configured to further accept input of information other than the input token sequence. Also, the inference model 5 may be configured to further output information other than the output token sequence.

[0111] §5 Working Examples In order to verify the effectiveness of the present invention, trained inference models according to the following examples and comparative examples were generated, and the inference accuracy of the generated trained inference models was evaluated.

[0112] Specifically, 261,396 samples of original music were prepared, and the motion-based tokenization method shown in Table 1 and FIGS. 6A and 8A was used to generate input token sequences constituting training data from each of the prepared sample music. Furthermore, assuming musical arrangement as the inference task, 261,396 samples of arranged music corresponding to each original music were prepared. Then, similar to the input token sequence, the motion-based tokenization method was used to generate true values ​​of output token sequences constituting correct labels from each sample of the arranged music. A training dataset of 261,396 samples was generated by associating the generated input token sequence (training data) with the true values ​​(correct labels) of the output token sequence. In the example, as shown in FIGS. 6A and 8A, a training dataset was obtained by placing metrical tokens at the positions of bar lines and beats for the true values ​​of the input token sequence and the output token sequence. In the comparative example, a training dataset was obtained without including metrical tokens in the true values ​​of the input token sequence and the output token sequence (otherwise the same as in the example).

[0113] The inference models according to the examples and comparative examples were structured using the Transformer illustrated in Fig. 9. Using a training dataset of 261,396 samples prepared in a similar manner to the above embodiment, machine learning was performed to generate trained inference models according to the examples and comparative examples.

[0114] In addition to the training data, 1000 samples of music (each sample having a duration of four bars) were prepared, and 1000 samples of input token sequences (target data) were obtained from the prepared music. As with the learning dataset, in the input token sequence of the example, each metrical token was placed at the position of a bar line and a beat. On the other hand, in the input token sequence of the comparative example, no metrical token was placed (otherwise the same as in the example).

[0115] Next, using the trained inference models for the Example and Comparative Example, an inference task was performed on the target data for each sample, and an output token sequence indicating the inference results was obtained. Then, it was evaluated whether or not a discrepancy in the number of beats occurred in the arrangement indicated by the output token sequence compared to the original music (i.e., the music indicated by the target data). As a result, a discrepancy in the number of beats occurred with a probability of 17.4% in the Comparative Example. On the other hand, a discrepancy in the number of beats occurred with a probability of 4.1% in the Example. These results demonstrate that the probability of temporal errors can be significantly reduced by incorporating metrical tokens indicating metrical structure. [Explanation of symbols]

[0116] 1...Model generation device, 11...control unit, 12...storage unit, 13...communication interface, 14...External interface, 15...input device, 16...output device, 17...drive, 81...model generation program, 91...storage medium, 111... learning data acquisition unit, 112... learning processing unit, 113...storage processing unit, 125...learning result data, 2...Music inference device, 21...control unit, 22...storage unit, 23...communication interface, 24...External interface, 25...input device, 26...output device, 27...drive, 82...Music inference program, 92...Storage medium, 211...data acquisition unit, 212...inference unit, 213...output unit, 221...Target data, 3...training dataset, 31...Training data, 32...Correct label, 5...Inference model

Claims

1. a data acquisition unit that acquires target data including an input token sequence generated from the musical piece using a motion-based or note-based tokenization scheme, the input token sequence being arranged to represent at least a portion of a musical note sequence of the musical piece, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions of the musical piece; an inference unit that uses a trained inference model to generate, as a result of inference on the music piece from the input token sequence included in the target data, an output token sequence generated by the action-based or note-based tokenization method, which output token sequence indicates a result of estimating local attributes of at least a portion of the music piece; an output unit that outputs the result of the inference; Equipped with The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a training piece of music, the input token sequence being generated from the training piece of music by the action-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme, the truth value corresponding to a result of estimating the local attributes of at least a portion of the training piece as a result of inference on the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference device.

2. a data acquisition unit configured to acquire target data including an input token sequence arranged to represent at least a portion of a piece of music and generated from the piece of music using a motion-based or note-based tokenization scheme, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions in the piece of music; an inference unit that generates, from the input token sequence included in the target data by using a trained inference model, an output token sequence generated by the action-based or note-based tokenization method, which output token sequence indicates an inference result for the music piece; an output unit that outputs the result of the inference; Equipped with each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the training piece of music by the movement-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method corresponding to a result of inference for the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference device.

3. each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; The music estimation device according to claim 1 .

4. The local attributes are: code, key, Beat, and Timing at which at least one of the chord, the key, and the time signature changes At least one of The music estimation device according to claim 1 .

5. The computer acquiring target data including an input token sequence arranged to represent a sequence of notes of at least a portion of a musical piece, the input token sequence being generated from the musical piece using a motion-based or note-based tokenization scheme, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions of the musical piece; generating an output token sequence generated by the action-based or note-based tokenization method from the input token sequence included in the target data using a trained inference model, the output token sequence indicating an estimated local attribute of at least a portion of the music piece as a result of inference on the music piece; outputting the result of the inference; Run The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a training piece of music, the input token sequence being generated from the training piece of music by the action-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme, the truth value corresponding to a result of estimating the local attributes of at least a portion of the training piece as a result of inference on the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference method.

6. On the computer, acquiring target data including an input token sequence arranged to represent a sequence of notes of at least a portion of a musical piece, the input token sequence being generated from the musical piece using a motion-based or note-based tokenization scheme, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions of the musical piece; generating an output token sequence generated by the action-based or note-based tokenization method from the input token sequence included in the target data using a trained inference model, the output token sequence indicating an estimated local attribute of at least a portion of the music piece as a result of inference on the music piece; outputting the result of the inference; Execute The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a training piece of music, the input token sequence being generated from the training piece of music by the action-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme, the truth value corresponding to a result of estimating the local attributes of at least a portion of the training piece as a result of inference on the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference program.

7. The computer acquiring target data including an input token sequence arranged to represent at least a portion of a musical piece and generated from the musical piece using a movement-based or note-based tokenization scheme, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions in the musical piece; generating an output token sequence generated by the movement-based or note-based tokenization scheme from the input token sequence included in the target data using a trained inference model, the output token sequence indicating an inference result for the music piece; outputting the result of the inference; Run each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the training piece of music by the movement-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method corresponding to a result of inference for the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference method.

8. On the computer, acquiring target data including an input token sequence arranged to represent at least a portion of a musical piece and generated from the musical piece using a movement-based or note-based tokenization scheme, the input token sequence including a plurality of metrical tokens arranged to represent metrical positions in the musical piece; generating an output token sequence generated by the movement-based or note-based tokenization scheme from the input token sequence included in the target data using a trained inference model, the output token sequence indicating an inference result for the music piece; outputting the result of the inference; Execute each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; The trained inference model is constructed by performing machine learning of the inference model using a plurality of learning data sets, each of which is composed of a combination of training data and a correct answer label; the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the training piece of music by the movement-based or note-based tokenization scheme; the input token sequence included in the training data includes a plurality of metrical tokens arranged to indicate metrical positions of the training piece of music; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method corresponding to a result of inference for the training piece; The machine learning is configured by training the inference model for each of the learning data sets so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label. Music inference program.

9. A learning data acquisition unit that acquires a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a musical piece to be learned, the input token sequence being generated from the musical piece using a motion-based or note-based tokenization method; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method, the truth value corresponding to a result of estimating local attributes of at least a portion of the music piece as a result of inference on the music piece. A learning data acquisition unit; a learning processing unit that performs machine learning of an inference model using the acquired multiple learning data sets, wherein the machine learning is configured by training the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; and Equipped with Model generation device.

10. A learning data acquisition unit that acquires a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the piece of music using a movement-based or note-based tokenization scheme; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme corresponding to a result of inference for the musical piece. A learning data acquisition unit; a learning processing unit that performs machine learning of an inference model using the acquired multiple learning data sets, wherein the machine learning is configured by training the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; and Equipped with each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; Model generation device.

11. each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; The model generating device according to claim 9.

12. The local attributes are: code, key, Beat, and Timing at which at least one of the chord, the key, and the time signature changes At least one of The model generating device according to claim 9.

13. The computer A step of obtaining a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a musical piece to be learned, the input token sequence being generated from the musical piece using a motion-based or note-based tokenization method; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method, the truth value corresponding to a result of estimating local attributes of at least a portion of the music piece as a result of inference on the music piece. Steps and a step of performing machine learning of an inference model using the acquired multiple learning data sets, the machine learning being configured to train the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; To execute Model generation method.

14. On the computer, A step of obtaining a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent a sequence of notes of at least a portion of a musical piece to be learned, the input token sequence being generated from the musical piece using a motion-based or note-based tokenization method; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization method, the truth value corresponding to a result of estimating local attributes of at least a portion of the music piece as a result of inference on the music piece. Steps and a step of performing machine learning of an inference model using the acquired multiple learning data sets, the machine learning being configured to train the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; Execute Model generator.

15. The computer A step of obtaining a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the piece of music using a movement-based or note-based tokenization scheme; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme corresponding to a result of inference for the musical piece. Steps and a step of performing machine learning of an inference model using the acquired multiple learning data sets, the machine learning being configured to train the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; Run each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; Model generation method.

16. On the computer, A step of obtaining a plurality of learning data sets each composed of a combination of training data and a correct answer label, the training data includes an input token sequence arranged to represent at least a portion of a training piece of music and generated from the piece of music using a movement-based or note-based tokenization scheme; the input token sequence includes a plurality of metrical tokens arranged to indicate metrical positions of the musical piece; the ground truth label is configured to indicate a truth value of an output token sequence generated by the movement-based or note-based tokenization scheme corresponding to a result of inference for the musical piece. Steps and a step of performing machine learning of an inference model using the acquired multiple learning data sets, the machine learning being configured to train the inference model for each learning data set so that an output token sequence generated by the inference model from the input token sequence included in the training data matches a true value indicated by the correct answer label; Execute each of the plurality of metrical tokens is placed at a position of a bar line and a beat of the musical piece in the input token sequence; Model generator.

Images