Information processing method, information processing system, electronic musical instrument, and program

Abstract

To specify suitable tones for new music.SOLUTION: An information processing system includes: a first acquisition unit 51 for acquiring acoustic data F indicating sound of target music; and a generation unit 53 for outputting tone data Z indicating tones suitable for target music from a learned model 60 by inputting input data X including acoustic data F to the learned model 60 that has learned a relation between sound of reference music and the tones used for the reference music.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for processing information about music. [Background technology]

[0002] Electronic musical instruments that can play back a piece of music played by a user in one of a number of different tones have been proposed. For example, Patent Document 1 discloses a technique that allows a user to set a tone suitable for a piece of music to be played. The tone suitable for a piece of music is registered in advance for each piece of music. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2007-140308 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology of Patent Document 1, it is necessary to register a tone color for each piece of music in advance. Therefore, it is not possible to set a suitable tone color for a new piece of music that has not been registered, such as a piece of music created by a user. In consideration of the above circumstances, one aspect of the present disclosure has an object to specify a tone color for a new piece of music. [Means for solving the problem]

[0005] In order to solve the above problems, an information processing system according to one embodiment of the present disclosure includes a first acquisition unit that acquires acoustic data representing the acoustics of a target song, and a generation unit that outputs tone data indicating a tone adapted to the target song from a trained model that has learned the relationship between the acoustics of a reference song and the tone used in the reference song by inputting input data including the acquired acoustic data into the trained model.

[0006] An electronic musical instrument according to one embodiment of the present disclosure includes a performance receiving unit that receives a performance by a user, a first acquisition unit that acquires acoustic data representing the acoustics of a target song, a generation unit that inputs input data including the acquired acoustic data into a trained model that has learned the relationship between the acoustics of a reference song and the tones used in the reference song, and outputs timbre data indicating a timbre adapted to the target song from the trained model, and a playback unit that plays back acoustics of a timbre corresponding to the timbre data in accordance with the performance received by the performance receiving unit.

[0007] An information processing method according to one aspect of the present disclosure acquires acoustic data representing the acoustics of a target song, inputs input data including the acquired acoustic data into a trained model that has learned the relationship between the acoustics of a reference song and the timbre used in the reference song, and outputs timbre data indicating a timbre adapted to the target song from the trained model.

[0008] A program according to one embodiment of the present disclosure causes a computer system to function as a first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music, and a generation unit that outputs tone data indicating a tone adapted to the target piece of music from a trained model that has learned the relationship between the acoustics of a reference piece of music and the tone used in the reference piece of music by inputting input data including the acquired acoustic data into the trained model. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a performance system in a first embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of an electronic musical instrument. [Diagram 3] FIG. 1 is a block diagram illustrating a configuration of an information processing system. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing system. [Diagram 5] FIG. 1 is a block diagram illustrating an example of the configuration of a trained model. [Figure 6] 11 is a flowchart illustrating a specific procedure of an analysis process. [Figure 7] 10 is a flowchart illustrating a specific procedure of a performance process. [Figure 8] FIG. 1 is a block diagram illustrating a configuration of a machine learning system. [Figure 9] FIG. 1 is a block diagram illustrating an example of the functional configuration of a machine learning system. [Figure 10] 11 is a flowchart illustrating a specific procedure of a learning process. [Figure 11] FIG. 11 is a block diagram illustrating a functional configuration of an information processing system according to a second embodiment. [Figure 12] FIG. 2 is a schematic diagram of reference data. [Figure 13] 11 is a flowchart illustrating a specific procedure of an estimation process. [Figure 14] FIG. 13 is a schematic diagram of a selection screen. [Figure 15] 10 is a flowchart illustrating a specific procedure of a control process. [Figure 16] FIG. 11 is a block diagram illustrating an example of the functional configuration of an electronic musical instrument according to a third embodiment. [Figure 17] FIG. 13 is a block diagram illustrating the configuration of a performance system according to a fourth embodiment. [Figure 18] FIG. 13 is a block diagram illustrating the configuration of a performance system in a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A: First embodiment 1 is a block diagram illustrating the configuration of a performance system 100 according to the first embodiment. The performance system 100 is a computer system for allowing a user U to play a desired piece of music (hereinafter referred to as a "target piece of music"), and includes a signal supplying device 10, an electronic musical instrument 20, an information processing system 30, and a machine learning system 40. The signal supplying device 10 is connected to the electronic musical instrument 20 by wire or wirelessly. The electronic musical instrument 20 and the information processing system 30 communicate with each other via a communication network 200 such as the Internet.

[0011] The signal supplying device 10 supplies an audio signal V to the electronic musical instrument 20. The audio signal V is a sample sequence representing the waveform of the performance sound of the target musical piece. For example, a playback device that supplies the audio signal V recorded on a recording medium such as a CD to the electronic musical instrument 20, or a communication device that supplies the audio signal V received from a distribution device (not shown) via a communication network 200 to the electronic musical instrument 20 is exemplified as the signal supplying device 10. For example, an information terminal such as a smartphone or a tablet terminal is also used as the signal supplying device 10. In addition, a sound collecting device that generates the audio signal V by collecting surrounding sounds is also used as the signal supplying device 10. The sound collecting device collects, for example, musical sounds produced by an instrument when played by a user U, or sounds produced by the user U when singing. The signal supplying device 10 may be mounted on the electronic musical instrument 20.

[0012] The electronic musical instrument 20 is a performance device used by a user U to play a target piece of music. An audio signal V provided from a signal supplying device 10 is transmitted from the electronic musical instrument 20 to an information processing system 30. The information processing system 30 generates accompaniment data C and timbre data Z by analyzing the audio signal V. The accompaniment data C is data indicating an accompaniment pattern P suitable for the target piece of music. For example, identification information for identifying one of a plurality of different accompaniment patterns P is generated as the accompaniment data C. Each accompaniment pattern P is a signal representing an accompaniment sound. For example, a rhythm pattern composed of musical sounds of a percussion instrument such as a drum is one example of an accompaniment pattern P.

[0013] The timbre data Z is data indicating a timbre suitable for the target piece of music. For example, identification information for identifying one of a plurality of different timbres is generated as the timbre data Z. For example, the timbre data Z indicates one of a plurality of timbres corresponding to different types of musical instruments (e.g., piano, violin, guitar, etc.). The timbre data Z may also indicate one of a plurality of timbres produced by different playing styles for one type of musical instrument. For example, for a stringed instrument, the timbre data Z indicates one of a plurality of timbres corresponding to different playing styles, such as plucked string playing and bowed string playing.

[0014] The accompaniment data C and timbre data Z generated by the information processing system 30 are transmitted to the electronic musical instrument 20 which is the source of the audio signal V. The electronic musical instrument 20 executes in parallel a process of reproducing the accompaniment sounds of the accompaniment pattern P indicated by the accompaniment data C, and a process of reproducing performance sounds of the timbre indicated by the timbre data Z in response to a performance by the user U. As can be understood from the above explanation, the user U can play a desired target musical piece with a timbre suitable for the target musical piece, in parallel with the reproduction of the accompaniment pattern P suitable for the target musical piece.

[0015] 2 is a block diagram illustrating an example of the configuration of an electronic musical instrument 20. The electronic musical instrument 20 is realized by a computer system including a control device 21, a storage device 22, a communication device 23, a performance device 24, an operation device 25, a display device 26, a sound source device 27, and a sound emission device 28. The electronic musical instrument 20 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0016] The control device 21 is composed of one or more processors that control each element of the electronic musical instrument 20. For example, the control device 21 is composed of one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0017] The storage device 22 is one or more memories that store the programs executed by the control device 21 and various data used by the control device 21. The storage device 22 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that the storage device 22 may be a portable recording medium that is detachable from the electronic musical instrument 20, or a recording medium (e.g., cloud storage) to which the control device 21 can write or read via, for example, the communication network 200. Note that the audio signal V may be stored in the storage device 22.

[0018] The storage device 22 of the first embodiment stores a plurality of different accompaniment patterns P. For example, musical elements such as time (e.g., 4 / 4 time, 3 / 4 time), type of instrument, rhythm, etc. differ for each accompaniment pattern P. Among the plurality of accompaniment patterns P stored in the storage device 22, an accompaniment pattern P indicated by the accompaniment data C received from the information processing system 30 is selectively played back. In other words, an accompaniment pattern P suitable for the target music piece is played back.

[0019] The communication device 23 communicates with the information processing system 30 via the communication network 200. Specifically, the communication device 23 transmits an audio signal V of the target music piece to the information processing system 30. The communication device 23 also receives accompaniment data C and timbre data Z transmitted from the information processing system 30. The communication line between the electronic musical instrument 20 and the information processing system 30 may or may not include a wireless section. The communication device 23, which is separate from the electronic musical instrument 20, may be connected to the electronic musical instrument 20 by wire or wirelessly. As the communication device 23 separate from the electronic musical instrument 20, for example, an information terminal such as a smartphone or a tablet terminal is used.

[0020] The performance device 24 is an input device that accepts a performance by the user U. For example, the performance device 24 is equipped with a keyboard on which a plurality of keys corresponding to different pitches are arranged. The user U plays the target piece of music by sequentially operating the desired keys on the performance device 24. Note that the specific form of the performance device 24 is arbitrary and is not limited to a keyboard. The performance device 24 is an example of a "performance accepting unit."

[0021] The operation device 25 is an input device that receives instructions from the user U. The operation device 25 is, for example, a plurality of operators operated by the user U, or a touch panel that detects contact by the user U. The display device 26 displays images under the control of the control device 21. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device 26.

[0022] The sound source device 27 generates a performance signal A according to a performance on the performance device 24. The performance signal A is an audio signal representing a waveform of a performance sound instructed by a performance on the performance device 24. Specifically, the sound source device 27 generates a performance signal A representing a musical tone of a pitch corresponding to a key pressed by the user U among the multiple keys of the performance device 24. The timbre of the performance sound represented by the performance signal A is variably set to any one of a plurality of timbres. For example, the sound source device 27 generates a performance signal A of a timbre indicated by timbre data Z received from the information processing system 30.

[0023] The sound source device 27 can generate a performance signal A which is a mixture of musical sounds corresponding to the performance on the performance device 24 and accompaniment sounds represented by the accompaniment pattern P. The control device 21 may realize the function of the sound source device 27 by executing a program stored in the storage device 22. In other words, the sound source device 27 dedicated to generating the performance signal A is omitted. The performance signal A generated by the sound source device 27 may be transmitted to the information processing system 30 as an audio signal V.

[0024] The sound emitting device 28 emits a performance sound represented by the performance signal A. For example, a speaker or headphones are used as the sound emitting device 28. As can be understood from the above description, when the electronic musical instrument 20 receives the accompaniment data C and the timbre data Z from the information processing system 30, the user U can play the target piece of music with the timbre indicated by the timbre data Z in parallel with the playback of the accompaniment pattern P indicated by the accompaniment data C. As can be understood from the above description, the sound source device 27 and the sound emitting device 28 in the first embodiment function as a playback unit 29 that plays back the sound of the timbre indicated by the timbre data Z in response to the performance by the user U.

[0025] 3 is a block diagram illustrating an example of the configuration of an information processing system 30. The information processing system 30 is realized by a computer system including a control device 31, a storage device 32, and a communication device 33. The information processing system 30 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0026] The control device 31 is composed of one or more processors that control each element of the information processing system 30. For example, the control device 31 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC.

[0027] The storage device 32 is one or more memories that store the programs executed by the control device 31 and various data used by the control device 31. The storage device 32 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that a portable recording medium that is detachable from the information processing system 30, or a recording medium (e.g., cloud storage) that the control device 31 can write to or read from via the communication network 200 may be used as the storage device 32.

[0028] The communication device 33 communicates with the electronic musical instrument 20 via the communication network 200. Specifically, the communication device 33 receives an audio signal V transmitted from the electronic musical instrument 20. The communication device 33 also transmits accompaniment data C and timbre data Z to the electronic musical instrument 20.

[0029] 4 is a block diagram illustrating an example of a functional configuration of the information processing system 30. The control device 31 of the information processing system 30 functions as an analysis processing unit 50 by executing a program stored in the storage device 32. The analysis processing unit 50 generates accompaniment data C and timbre data Z by analyzing the audio signal V. The analysis processing unit 50 includes a first acquisition unit 51, a second acquisition unit 52, and a generation unit 53.

[0030] The first acquisition unit 51 acquires audio data F representing the performance sound of the target music piece. Specifically, the first acquisition unit 51 generates the audio data F by analyzing the audio signal V. For example, the audio data F is generated from a part or all of the audio signal V. The audio data F is data in an arbitrary format representing a temporal change in the performance sound of the target music piece. Specifically, the audio data F is data representing a time series of the frequency characteristics of the performance sound. For example, data representing a time series of frequency characteristics such as MFCC (Mel-Frequency Cepstrum Coefficient), MSLS (Mel-Scale Log Spectrum), or CQT (Constant-Q Transform) is exemplified as the audio data F. The audio data F is also expressed as data representing the features (timbre features) related to the timbre of the performance sound represented by the audio signal V. For example, a known frequency analysis such as a short-time Fourier transform is arbitrarily adopted to generate the audio data F. Note that a sample series constituting the audio signal V may be used as the audio data F.

[0031] The second acquisition unit 52 acquires accompaniment data C indicating an accompaniment pattern P suitable for the target music piece. Specifically, the second acquisition unit 52 generates the accompaniment data C by analyzing the audio signal V. The accompaniment data C is generated from a part or all of the audio signal V. Specifically, the first acquisition unit 51 generates the audio data F and the second acquisition unit 52 generates the accompaniment data C by using a part of the audio signal V. For example, the second acquisition unit 52 first estimates the music genre of the target music piece by analyzing the audio signal V. For example, a known technique such as JP 2015-79110 A or the like is arbitrarily adopted for estimating the music genre. Then, the second acquisition unit 52 identifies the accompaniment data C corresponding to the estimated music genre of the target music piece among the multiple accompaniment data C corresponding to different music genres. Note that the second acquisition unit 52 may identify the accompaniment data C by analyzing the audio data F. The multiple accompaniment data C that are candidates for acquisition by the second acquisition unit 52 are stored in, for example, the storage device 32.

[0032] The generating unit 53 generates timbre data Z according to input data X including the audio data F acquired by the first acquiring unit 51 and the accompaniment data C acquired by the second acquiring unit 52. Specifically, the generating unit 53 generates timbre data Z indicating a suitable timbre for a combination of a performance sound represented by the audio signal V and an accompaniment pattern P specified by the accompaniment data C. The generating unit 53 uses the trained model 60 to generate the timbre data Z. There is no consideration as to whether the timbre of the performance sound represented by the audio signal V is different from the timbre indicated by the timbre data Z.

[0033] There is a correlation between the temporal change in the performance sound of a piece of music (acoustic data F) and the tone color frequently used in the performance of the piece of music. There is also a correlation between the accompaniment pattern P suitable for a piece of music and the tone color frequently used in the performance of the piece of music. The trained model 60 is a statistical estimation model that has learned the above tendencies. That is, the trained model 60 is a statistical estimation model that has learned the relationship between the combination of the performance sound and the accompaniment pattern P in each of a plurality of known pieces of music (hereinafter referred to as "reference pieces of music") and the tone color frequently used in the reference pieces of music by machine learning. Specifically, the trained model is a statistical estimation model that has learned the relationship between the input data X of the reference piece of music (the combination of the acoustic data F and the accompaniment data C) and the tone color data Z of the reference piece of music. The generation unit 53 inputs the input data X including the acoustic data F and the accompaniment data C to the trained model 60, and outputs the tone color data Z from the trained model 60. The tone color frequently used in the reference piece of music can also be said to be a tone color adapted to the reference piece of music (a tone color suitable for the performance of the reference piece of music).

[0034] The trained model 60 is configured, for example, by a deep neural network (DNN). For example, any type of neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), is used as the trained model 60. The trained model 60 may be configured by combining multiple types of deep neural networks. In addition, additional elements, such as a long short-term memory (LSTM), may be installed in the trained model 60.

[0035] The trained model 60 is realized by a combination of a program that causes the control device 31 to execute a calculation to generate timbre data Z from input data X, and a plurality of variables that are applied to the calculation. The plurality of variables related to the trained model 60 include, for example, weights and biases. The program and the plurality of variables that realize the trained model 60 are stored in the storage device 32. The numerical values ​​of each of the plurality of variables that define the trained model 60 are set in advance by machine learning.

[0036] 5 is a block diagram illustrating a specific configuration of the trained model 60. The trained model 60 includes a first model 61, a second model 62, and a third model 63. Acoustic data F of the input data X is input to the first model 61, and accompaniment data C of the input data X is input to the second model 62.

[0037] The first model 61 generates first data y1 representing characteristics of the performance sound of the target music piece from the sound data F. The first model 61 is a trained model 60 that has learned the relationship between the sound data F and the first data y1. In other words, the first model 61 is a model that extracts the characteristics of the sound data F. The first data y1 is data representing characteristics of the sound data F that contribute to the trained model 60 generating timbre data Z suitable for the target music piece from the input data X.

[0038] For example, in a configuration in which the first model 61 is configured with a convolutional neural network, a time series of frequency characteristics represented by the sound data F (i.e., a group of values ​​distributed in the time-frequency domain) is input to the first model 61 as a two-dimensional image. In a configuration in which the first model 61 is configured with a recurrent neural network, portions of the sound data F corresponding to each time point on the time axis are input sequentially to the first model 61. In a configuration in which the first model 61 is configured with a combination of a convolutional neural network and a recurrent neural network, portions of the sound data F corresponding to each time point on the time axis are input sequentially to the convolutional neural network, and the output data for each time of the convolutional neural network is input sequentially to a subsequent recurrent neural network.

[0039] The second model 62 generates second data y2 representing the characteristics of the accompaniment pattern P from the accompaniment data C. The second model 62 is a trained model 60 that has trained the relationship between the accompaniment data C and the second data y2. In other words, the second model 62 is a model that converts the identification information of the accompaniment pattern P indicated by the accompaniment data C into the second data y2, and is configured, for example, by a convolutional neural network.

[0040] The second data y2 is, for example, an embedding vector set in a multi-dimensional virtual space. The virtual space is a continuous space in which the position of each accompaniment pattern P (i.e., the coordinates specified by the second data y2) is determined according to the acoustic characteristics of the accompaniment pattern P. The more similar the acoustic characteristics are between the multiple accompaniment patterns P, the smaller the distance between the coordinates specified in the virtual space by the second data y2 of each accompaniment pattern P becomes. In other words, the virtual space is expressed as a space that represents the relationship between the multiple accompaniment patterns P.

[0041] Intermediate data Y including the first data y1 generated by the first model 61 and the second data y2 generated by the second model 62 is input to a third model 63. The third model 63 is a trained model 60 that has trained the relationship between the intermediate data Y and timbre data Z, and generates the timbre data Z from the intermediate data Y. The third model 63 is configured, for example, by a recurrent neural network or a convolutional neural network.

[0042] 6 is a flow chart illustrating a specific procedure of a process (hereinafter referred to as an "analysis process") Sa in which the control device 31 generates the timbre data Z. For example, the analysis process Sa is started upon receipt of an audio signal V transmitted from the electronic musical instrument 20.

[0043] When the analysis process Sa is started, the first acquisition unit 51 generates audio data F by analyzing the audio signal V of the target music piece (Sa1). The second acquisition unit 52 generates accompaniment data C by analyzing the audio signal V (Sa2). The second acquisition unit 52 transmits the accompaniment data C from the communication device 33 to the electronic musical instrument 20 (Sa3). Note that the order of the generation of audio data F by the first acquisition unit 51 (Sa1) and the generation (Sa2) and transmission (Sa3) of accompaniment data C by the second acquisition unit 52 may be reversed.

[0044] The generation unit 53 inputs the input data X including the audio data F and the accompaniment data C to the trained model 60, thereby outputting the timbre data Z from the trained model 60 (Sa4). The generation unit 53 transmits the timbre data Z from the communication device 33 to the electronic musical instrument 20 (Sa5). Note that the accompaniment data C may be transmitted to the electronic musical instrument 20 together with the timbre data Z.

[0045] 7 is a flow chart showing a specific procedure of a process (hereinafter referred to as "performance process") Sb executed by the control device 21 of the electronic musical instrument 20 upon receiving the accompaniment data C and the timbre data Z. The performance process Sb is started upon receipt of the accompaniment data C and the timbre data Z.

[0046] When the performance process Sb is started, the control device 21 instructs the sound source device 27 to select the tone color indicated by the tone color data Z (Sb1). Therefore, the sound source device 27 is ready to generate a performance signal A representing a musical tone of the tone color indicated by the tone color data Z in response to an operation of the performance device 24 by the user U.

[0047] The user U can instruct the playback of the accompaniment pattern P by operating the operating device 25. The control device 21 waits until the user U instructs the playback of the accompaniment pattern P (Sb2: NO). When the user U instructs the playback of the accompaniment pattern P (Sb2: YES), the control device 21 instructs the sound source device 27 to play the accompaniment pattern P indicated by the accompaniment data C received from the information processing system 30 (Sb3).

[0048] The control device 21 judges whether the user U has operated the performance device 24 (Sb4). If the performance device 24 has been operated (Sb4: YES), the control device 21 instructs the sound source device 27 to produce a sound of a pitch corresponding to the key operated by the user U (Sb5). The sound source device 27 generates a performance signal A representing a performance sound of the timbre indicated by the timbre data Z. Thus, the performance sound by the user U and the accompaniment sound of the accompaniment pattern P are reproduced from the sound emitting device 28. If the performance device 24 has not been operated (Sb4: NO), the reproduction of the performance sound (Sb5) is not executed.

[0049] The control device 21 repeats the instruction (Sb4, Sb5) to the sound source device 27 to play the performance sound until the user U instructs the performance to end (Sb6: NO). When the user U instructs the performance to end (Sb6: YES), the control device 21 ends the performance process Sb.

[0050] According to the first embodiment exemplified above, by inputting input data X including acoustic data F representing the acoustics of a target piece of music into the trained model 60, timbre data Z indicating a timbre adapted to the target piece of music is generated. Therefore, for example, a suitable timbre can be identified for a new piece of music. In particular, in the first embodiment, input data X including acoustic data F representing the performance sounds of the target piece of music and accompaniment data C indicating an accompaniment pattern P corresponding to the target piece of music is inputted into the trained model 60. Therefore, a suitable timbre can be identified for the combination of the acoustics of the target piece of music and the accompaniment pattern P of the target piece of music.

[0051] Moreover, according to the first embodiment, even if the user U does not have the musical knowledge to select an accompaniment pattern P or a tone color suitable for the target music piece, there is an advantage that the user U can select an appropriate accompaniment pattern P and tone color. There is also an advantage that the effort required for the user U to select an appropriate accompaniment pattern P and tone color can be reduced.

[0052] The machine learning system 40 in Fig. 1 generates the trained model 60 exemplified above. Fig. 8 is a block diagram illustrating an example of the configuration of the machine learning system 40. The machine learning system 40 includes a control device 41, a storage device 42, and a communication device 43. Note that the machine learning system 40 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0053] The control device 41 is composed of one or more processors that control each element of the machine learning system 40. The control device 41 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The communication device 43 communicates with the information processing system 30.

[0054] The storage device 42 is a single or multiple memories that store the programs executed by the control device 41 and various data used by the control device 41. The storage device 42 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the machine learning system 40, or a recording medium (e.g., cloud storage) that can be written to or read by the control device 41 via the communication network 200 may be used as the storage device 42.

[0055] 9 is a block diagram illustrating an example of a functional configuration of the machine learning system 40. The control device 41 executes a program stored in the storage device 42 to function as multiple elements (a training data acquisition unit 71 and a learning processing unit 72) for establishing the trained model 60 by machine learning.

[0056] The learning processing unit 72 establishes a learned model 60 by supervised machine learning (learning processing Sc described later) using multiple pieces of training data T. The training data acquisition unit 71 acquires multiple pieces of training data T. Specifically, the training data acquisition unit 71 acquires, from the storage device 42, multiple pieces of training data T stored in the storage device 42.

[0057] Each of the multiple training data T is composed of a combination of training input data Xt and training timbre data Zt. The input data Xt includes audio data Ft of a known reference piece of music, and accompaniment data Ct of an accompaniment pattern P suitable for the reference piece of music. The audio data Ft is generated from the results of recording a performance of the reference piece of music. The accompaniment pattern P indicated by the accompaniment data Ct is selected, for example, by the creator of the training data T, taking into consideration the musical characteristics of the reference piece of music (e.g., melody or beat, etc.).

[0058] The timbre data Zt of each training data T is data indicating a timbre suitable for the reference music piece corresponding to the training data T. In other words, the timbre data Zt of each training data T corresponds to a correct answer value (label) for the input data Xt of the training data T. The timbre data Zt is selected, for example, by the creator of the training data T, taking into consideration the musical characteristics of the combination of the reference music piece and the accompaniment pattern P.

[0059] 10 is a flowchart illustrating a specific procedure of the learning process Sc in which the control device 41 establishes the trained model 60. The learning process Sc is also expressed as a method of generating the trained model 60 by machine learning (trained model generation method).

[0060] When the learning process Sc is started, the training data acquisition unit 71 acquires any one of the multiple training data T (hereinafter referred to as "selected training data T") stored in the storage device 42 (Sc1). As illustrated in Fig. 9, the learning processing unit 72 inputs the input data Xt of the selected training data T to an initial or provisional model (hereinafter referred to as "provisional model 65") (Sc2), and acquires the timbre data Z output by the provisional model 65 in response to the input (Sc3).

[0061] The learning processing unit 72 calculates a loss function that represents the error between the timbre data Z generated by the provisional model 65 and the timbre data Zt of the selected training data T (Sc4). The learning processing unit 72 updates multiple variables of the provisional model 65 so that the loss function is reduced (ideally minimized) (Sc5). For example, the backpropagation method is used to update the multiple variables according to the loss function.

[0062] The learning processing unit 72 determines whether a predetermined termination condition is satisfied (Sc6). The termination condition is, for example, that the loss function falls below a predetermined threshold or that the change amount of the loss function falls below a predetermined threshold. If the termination condition is not satisfied (Sc6: NO), the training data acquisition unit 71 selects the unselected training data T as new selected training data T (Sc1). That is, the process of updating multiple variables of the trained model 60 (Sc2-Sc5) is repeated until the termination condition is satisfied (Sc6: YES). If the termination condition is satisfied (Sc6: YES), the learning processing unit 72 terminates the update (Sc2-Sc5) of multiple variables that define the provisional model 65. The provisional model 65 at the time when the termination condition is satisfied is determined as the trained model 60. That is, the multiple variables of the trained model 60 are determined to be the numerical values ​​at the time when the learning process Sc is terminated.

[0063] As can be understood from the above explanation, the trained model 60 outputs statistically valid timbre data Z for unknown input data X, based on the latent relationship between the input data Xt and timbre data Zt in multiple training data T. In other words, as described above, the trained model 60 is a model that has learned, by machine learning, the relationship between the performance notes and accompaniment pattern P of a reference piece of music, and the timbres frequently used in the reference piece of music. The trained model 60 can also be expressed as a model that has learned the relationship between the input data Xt of a reference piece of music and the timbre data Zt of the reference piece of music.

[0064] The learning processing unit 72 transmits the trained model 60 established by the above procedure from the communication device 43 to the information processing system 30 (Sc7). Specifically, the learning processing unit 72 transmits multiple variables of the trained model 60 from the communication device 43 to the information processing system 30. The control device 31 of the information processing system 30 stores the trained model 60 received from the machine learning system 40 in the storage device 32. Specifically, multiple variables that define the trained model 60 are stored in the storage device 32.

[0065] B: Second embodiment A second embodiment will be described. Note that, for elements having the same functions as those in the first embodiment in each of the following exemplary aspects, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0066] 11 is a block diagram illustrating a functional configuration of an information processing system 30 in the second embodiment. In the first embodiment, a configuration in which the information processing system 30 receives an audio signal V from an electronic musical instrument 20 is illustrated. A communication device 33 of the information processing system 30 in the second embodiment receives the audio signal V or performance data D from the electronic musical instrument 20.

[0067] The audio signal V is a sample sequence representing the waveform of the performance sound of the target music piece, as in the first embodiment. On the other hand, the performance data D is time-series data representing the content of the performance of the user U on the performance device 24. Specifically, the performance data D is data in a format conforming to the MIDI (Musical Instrument Digital Interface) standard, for example, and specifies the pitch and sound duration of each of the multiple notes that make up the music piece.

[0068] The control device 31 of the second embodiment executes a program stored in the storage device 32, thereby functioning as an analysis processing unit 50 similar to that of the first embodiment, as well as a music estimation unit 56 and a timbre identification unit 57.

[0069] The music estimation unit 56 estimates the target music piece played by the user U by analyzing the audio signal V or performance data D received by the communication device 33 from the electronic musical instrument 20. Specifically, the music estimation unit 56 identifies a plurality of music pieces (hereinafter referred to as "candidate music pieces") that are highly likely to correspond to the target music piece. The music estimation unit 56 estimates the candidate music pieces using reference data R stored in the storage device 32. Note that the audio signal V and performance data D used to identify the candidate music pieces are collectively expressed as data representing the performance by the user U.

[0070] FIG. 12 is a schematic diagram of the reference data R. As illustrated in FIG. 12, the reference data R is a database in which music information Ra (Ra1, Ra2, ...), comparison data Rb (Rb1, Rb2, ...), accompaniment data C (C1, C2, ...), and timbre data Z (Z1, Z2, ...) are registered for each of a plurality of existing musical pieces. The music information Ra for each musical piece includes information such as the identification information and the name of the musical piece. The comparison data Rb for each musical piece is time-series data representing the contents of the musical piece. Specifically, like the performance data D, the comparison data Rb is data in a format conforming to, for example, the MIDI standard, and specifies the pitch and the sounding period for each of a plurality of notes constituting the musical piece. The accompaniment data C for each musical piece is data indicating an accompaniment pattern P suitable for the musical piece. Moreover, the timbre data Z for each musical piece is data indicating a timbre suitable for the musical piece.

[0071] 13 is a flowchart illustrating a specific procedure of a process (hereinafter, referred to as the "estimation process") Sd executed by the music estimation unit 56. For example, the estimation process Sd is started upon reception of an audio signal V or performance data D transmitted from the electronic musical instrument 20.

[0072] When the estimation process Sd is started, the music estimation unit 56 judges whether or not the communication device 33 has received the audio signal V from the electronic musical instrument 20 (Sd1). If the communication device 33 has received the audio signal V (Sd1: YES), the music estimation unit 56 generates performance data D from the audio signal V (Sd2). The performance data D is time-series data representing the content of the performance by the user U. Any known transcription technique may be used in the process of generating the performance data D from the audio signal V. If the communication device 33 has received the performance data D (Sd1: NO), the generation of the performance data D (Sd2) is not executed. As described above, the music estimation unit 56 acquires the performance data D representing the performance by the user U. The performance data D is data generated from the audio signal V received by the communication device 33, or data received by the communication device 33 from the electronic musical instrument 20.

[0073] The music estimation unit 56 compares the comparison data Rb and the performance data D registered for each piece of music in the reference data R to identify a predetermined number of candidate pieces of music (Sd3). Specifically, the music estimation unit 56 calculates the similarity between the comparison data Rb and the performance data D for each piece of music registered in the reference data R, and selects a predetermined number of pieces of music that are ranked high in descending order of similarity from among the pieces of music as candidate pieces of music. That is, the music estimation unit 56 identifies a predetermined number of candidate pieces of music whose comparison data Rb is similar to the performance data D. Therefore, the candidate pieces of music are pieces of music that correspond to the pieces of music played by the user U. The music estimation unit 56 transmits the music information Ra registered in the reference data R for each of the candidate pieces of music identified by the above process from the communication device 33 to the electronic musical instrument 20 (Sd4). That is, the music information Ra of the plurality of candidate pieces of music is transmitted to the electronic musical instrument 20.

[0074] The control device 21 of the electronic musical instrument 20 causes the display device 26 to display the music information Ra of each candidate music piece received from the information processing system 30. FIG. 14 is a schematic diagram of a screen (hereinafter referred to as the "selection screen") G displayed on the display device 26. The display device 26 displays the music information Ra (specifically, the music title) for each of the multiple candidate music pieces, as well as the character string "unregistered music piece", which means that the music piece is not registered in the reference data R. An unregistered music piece is, for example, a music piece that has been independently created by the user U. The user U selects a target music piece from the selection screen G by operating the operation device 25. The control device 21 transmits a selection instruction E indicating the target music piece selected by the user U from the communication device 23 to the information processing system 30.

[0075] For example, if the target song is an existing candidate song, the user U selects the candidate song from the selection screen G. When the user U selects a candidate song, the control device 21 transmits a selection instruction E indicating the candidate song from the communication device 33 to the information processing system 30. For example, the selection instruction E including the song information Ra of the candidate song is transmitted to the information processing system 30. On the other hand, if the target song is a song other than the candidate song (for example, a song composed by the user), the user U selects an unregistered song from the selection screen G. When the user U selects an unregistered song, the control device 21 transmits a selection instruction E indicating that the target song is an unregistered song from the communication device 33 to the information processing system 30.

[0076] For existing candidate musical pieces, accompaniment data C and timbre data Z are registered in the reference data R, whereas for unregistered musical pieces such as user U's own compositions, accompaniment data C and timbre data Z are not registered in the reference data R. When a target musical piece is a candidate musical piece registered in the reference data R, the timbre identification unit 57 in FIG. 11 identifies the accompaniment data C and timbre data Z of the candidate musical piece from the reference data R. Specifically, the timbre identification unit 57 acquires the accompaniment data C and timbre data Z registered for the target musical piece from the storage device 32. On the other hand, when the target musical piece is an unregistered musical piece, the analysis processing unit 50 of the second embodiment generates the accompaniment data C and timbre data Z of the target musical piece by analyzing the audio signal V. The configuration and operation of the analysis processing unit 50 are the same as those of the first embodiment.

[0077] 15 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as a "control process") Se executed by the control device 31 of the information processing system 30 in the second embodiment. For example, the control process Se is started in response to reception of an audio signal V or performance data D.

[0078] When the control process Se is started, the music estimation unit 56 executes the estimation process Sd illustrated in Fig. 13. That is, the music estimation unit 56 notifies the electronic musical instrument 20 of a predetermined number of candidate music pieces whose comparison data Rb are similar to the performance data D. After executing the estimation process Sd, the control device 31 waits until the communication device 33 receives a selection instruction E from the electronic musical instrument 20 (Se1: NO).

[0079] When the communication device 33 receives a selection instruction E (Se1: YES), the control device 31 determines whether or not the selection instruction E indicates a candidate song (Se2). The above determination is a process of determining whether or not the target song is registered in the reference data R. In other words, when the selection instruction E indicates a candidate song (Se2: YES), it means that the target song is registered in the reference data R, and when the selection instruction E indicates an unregistered song (Se2: NO), it means that the target song is not registered in the reference data R. If the target song is not registered in the reference data R, the accompaniment data C and timbre data Z are not identified from the reference data R.

[0080] If the selection instruction E indicates a candidate song (Se2: YES), the timbre identification unit 57 identifies the accompaniment data C and timbre data Z of the candidate song from the reference data R (Se3). The timbre identification unit 57 transmits the accompaniment data C and timbre data Z of the candidate song from the communication device 33 to the electronic musical instrument 20 (Se4). The electronic musical instrument 20 executes the performance process Sb of FIG. 7 using the accompaniment data C and timbre data Z received from the information processing system 30.

[0081] On the other hand, if the selection instruction E indicates an unregistered song (Se2: NO), the analysis processing unit 50 executes the analysis process Sa of Fig. 6 using the trained model 60. That is, the analysis processing unit 50 generates accompaniment data C and timbre data Z by analyzing the audio signal V received from the electronic musical instrument 20, and transmits the accompaniment data C and timbre data Z from the communication device 33 to the electronic musical instrument 20 (Sa3, Sa5). The electronic musical instrument 20 executes the performance process Sb of Fig. 7 using the accompaniment data C and timbre data Z received from the information processing system 30.

[0082] As described above, in the second embodiment, when the timbre (timbre data Z) of the target music piece is registered in the reference data R, the timbre identification unit 57 identifies the timbre data Z from the reference data R, and when the timbre of the target music piece is not registered in the reference data R, the analysis processing unit 50 generates the timbre data Z using the trained model 60. That is, when an appropriate timbre for the target music piece is registered in the reference data R, the timbre data Z indicating the registered timbre is generated. Therefore, while identifying an appropriate timbre for a registered music piece, it is also possible to identify an appropriate timbre for an unregistered music piece (for example, a new music piece created by the user U). In addition, since the accompaniment data C and the timbre data Z are identified from the reference data R for a registered target music piece, the analysis process Sa is not required. Therefore, there is also an advantage in that the load required for the analysis process Sa is reduced.

[0083] C: Third embodiment 16 is a block diagram illustrating a functional configuration of an electronic musical instrument 20 according to a third embodiment. In the second embodiment, a configuration was exemplified in which the information processing system 30 includes an analysis processing unit 50, a music estimation unit 56, and a timbre identification unit 57. In the third embodiment, the electronic musical instrument 20 includes an analysis processing unit 50, a music estimation unit 56, and a timbre identification unit 57. The above elements are realized by the control device 21 executing a program stored in the storage device 22.

[0084] 16 (music estimation unit 56, timbre identification unit 57, and analysis processing unit 50) are similar in specific configuration and operation to those of the first and second embodiments. For example, the music estimation unit 56 analyzes the audio signal V supplied from the signal supply device 10, or the performance data D corresponding to the performance on the performance device 24, to identify multiple candidate songs that are highly likely to be the target song played by the user U. The music estimation unit 56 displays a selection screen G including music information Ra of each candidate song on the display device 26, and accepts an operation on the selection screen G from the user U.

[0085] When the target musical piece is a candidate musical piece registered in the reference data R, the timbre identification unit 57 identifies the accompaniment data C and timbre data Z of the candidate musical piece from the reference data R. The reference data R applied to the estimation of the candidate musical piece by the musical piece estimation unit 56 and the processing by the timbre identification unit 57 is stored in the storage device 22.

[0086] If the target piece of music is an unregistered piece of music, the analysis processing unit 50 generates accompaniment data C and timbre data Z of the target piece of music by analyzing the audio signal V. The trained model 60 applied to the analysis process Sa by the analysis processing unit 50 is stored in the storage device 22. In other words, the trained model 60 generated by the machine learning system 40 is transferred to the electronic musical instrument 20. The configuration of the trained model 60 is the same as in the first embodiment.

[0087] As in the first embodiment, the sound source device 27 generates a performance signal A representing a mixed sound of the accompaniment sound of the accompaniment pattern P indicated by the accompaniment data C and the performance sound of the timbre indicated by the timbre data Z. The performance signal A is supplied to a sound emitting device 28. Thus, the sound source device 27 and the sound emitting device 28 function as a reproduction unit 29 that reproduces the sound of the timbre indicated by the timbre data Z in response to the performance by the user U and reproduces the accompaniment sound of the accompaniment pattern P indicated by the accompaniment data C.

[0088] As will be understood from the above description, the third embodiment also achieves the same effects as the first and second embodiments. Note that, although the electronic musical instrument 20 including the analysis processing unit 50, the music estimation unit 56, and the timbre identification unit 57 is illustrated in Fig. 16, the music estimation unit 56 and the timbre identification unit 57 may be omitted from the electronic musical instrument 20.

[0089] D: Fourth embodiment 17 is a block diagram illustrating the configuration of a performance system 100 according to the fourth embodiment. The performance system 100 includes an electronic musical instrument 20 and an information device 80. The information device 80 is, for example, a smartphone or a tablet terminal. The information device 80 is connected to the electronic musical instrument 20, for example, by wire or wirelessly.

[0090] The information device 80 is realized by a computer system including a control device 81 and a storage device 82. The control device 81 is composed of one or more processors that control each element of the information device 80. For example, the control device 81 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The storage device 82 is one or more memories that store programs executed by the control device 81 and various data used by the control device 81. The storage device 82 is composed of a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that a portable recording medium that is detachable from the information device 80, or a recording medium (e.g., cloud storage) that the control device 81 can write or read via the communication network 200, may be used as the storage device 82.

[0091] The control device 81 executes a program stored in the storage device 82 to realize the analysis processing unit 50, the music estimation unit 56, and the timbre identification unit 57. The configurations and operations of the analysis processing unit 50, the music estimation unit 56, and the timbre identification unit 57 are similar to those illustrated in the first to third embodiments. Reference data R used by the music estimation unit 56 and the timbre identification unit 57, and the trained model 60 used by the analysis processing unit 50 are stored in the storage device 82.

[0092] The accompaniment data C and timbre data Z identified by the analysis processing unit 50 or the music estimation unit 56 are transmitted to the electronic musical instrument 20. As in the first embodiment, the sound source device 27 of the electronic musical instrument 20 generates a performance signal A representing a mixed sound of the accompaniment sound of the accompaniment pattern P indicated by the accompaniment data C and the performance sound of the timbre indicated by the timbre data Z. The performance signal A is supplied to a sound emitting device 28.

[0093] As can be understood from the above description, the fourth embodiment also achieves the same effects as the first to third embodiments. Note that, although an information device 80 including an analysis processing unit 50, a music estimation unit 56, and a timbre identification unit 57 is illustrated in Fig. 17, the music estimation unit 56 and the timbre identification unit 57 may be omitted from the information device 80.

[0094] In the fourth embodiment, for example, the trained model 60 constructed by the machine learning system 40 is transferred to the information device 80, and the trained model 60 is stored in the storage device 82. In the above configuration, the machine learning system 40 may be equipped with an authentication processing unit that authenticates the legitimacy of the user of the information device 80 (that is, that the user is a regular user registered in advance). When the legitimacy of the user is authenticated by the authentication processing unit, the trained model 60 is automatically transferred to the information device 80 (i.e., without requiring any instruction from the user).

[0095] E: Fifth embodiment 18 is a block diagram illustrating the configuration of a performance system 100 according to the fifth embodiment. As in the fourth embodiment, the performance system 100 includes an electronic musical instrument 20 and an information device 80. The configurations of the electronic musical instrument 20 and the information device 80 are the same as in the fourth embodiment.

[0096] The machine learning system 40 stores a plurality of trained models 60 corresponding to different models of electronic musical instruments 20. The types and total number of tones that the electronic musical instrument 20 (specifically, the sound source device 27) can produce differ for each model of electronic musical instrument 20. The trained model 60 corresponding to each model outputs timbre data Z of tones that the electronic musical instrument 20 of that model can produce, and does not output timbre data Z of tones that the electronic musical instrument 20 of that model cannot produce. Specifically, in the learning process Sc for establishing the trained model 60 for each model, training data T including timbre data Zt of tones that the electronic musical instrument 20 of that model can produce is used. That is, a set of a plurality of training data T is individually prepared for each model of electronic musical instrument 20 (i.e., for each combination of tones that can be produced), and the trained model 60 is established by an individual learning process Sc for each model.

[0097] The information device 80 selectively acquires one of the multiple trained models 60 held by the machine learning system 40 via the communication network 200. Specifically, the information device 80 acquires one of the multiple trained models 60 from the machine learning system 40 that corresponds to the model of the electronic musical instrument 20 connected to the information device 80. The trained model 60 acquired from the machine learning system 40 is stored in the storage device 82 and used in the analysis process Sa by the analysis processing unit 50. The specific procedure of the analysis process Sa is the same as in each of the above-mentioned forms.

[0098] As can be understood from the above description, the fifth embodiment also achieves the same effects as the first to third embodiments. In addition, in the fifth embodiment, a trained model 60 is established for each model of electronic musical instrument 20, and therefore, compared to a configuration in which a common trained model 60 is used regardless of the model of electronic musical instrument 20, there is an advantage in that the timbre data Z appropriate for each model of electronic musical instrument 20 can be estimated with high accuracy. Note that the information processing system 30 of the first or second embodiment, the electronic musical instrument 20 of the third embodiment, and the information device 80 of the fourth or fifth embodiment are examples of an "information processing system."

[0099] F: Variation Specific modified embodiments that can be added to each of the above-mentioned embodiments are illustrated below. A plurality of embodiments arbitrarily selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0100] (1) The analysis processing unit 50 may generate timbre data Z for each period of a predetermined length on the time axis (hereinafter referred to as a "unit period"). Specifically, the first acquisition unit 51 generates sound data F for each unit period of the sound signal V. The generation unit 53 generates the timbre data Z for each of a plurality of unit periods from input data X including the sound data F for that unit period and accompaniment data C identified by the second acquisition unit 52.

[0101] The control device 21 or 31 may specify one type of timbre to be set in the sound source device 27 from a plurality of timbre data Z corresponding to different unit periods. For example, the most frequent timbre may be selected from the plurality of timbre data Z. Alternatively, a predetermined number of timbres may be presented to the user U in descending order of the number of times specified by the generation unit 53, and the user U may select one type of timbre to be set in the sound source device 27 from the predetermined number of timbres.

[0102] (2) In the first embodiment, the first acquisition unit 51 generates the acoustic data F, but the first acquisition unit 51 may receive the acoustic data F generated from the acoustic signal V in the electronic musical instrument 20 from the electronic musical instrument 20. That is, the acquisition of the acoustic data F by the first acquisition unit 51 includes both the generation of the acoustic data F and the reception of the acoustic data F. In the first embodiment, the second acquisition unit 52 generates the accompaniment data C, but the second acquisition unit 52 may receive the accompaniment data C generated from the acoustic signal V in the electronic musical instrument 20 from the electronic musical instrument 20. That is, the acquisition of the accompaniment data C by the second acquisition unit 52 includes both the generation of the accompaniment data C and the reception of the accompaniment data C. In addition, the first acquisition unit 51 or the second acquisition unit 52 may be mounted on the electronic musical instrument 20.

[0103] (3) In the above-mentioned embodiments, the timbre data Z indicates one type of timbre, but the format of the timbre data Z is not limited to the above examples. For example, the timbre data Z may be data indicating a probability distribution for each of a plurality of types of timbre. Specifically, the timbre data Z specifies, for each timbre, the mean and variance of a probability distribution expressed, for example, as a normal distribution. The control device 21 or the control device 31 identifies the most likely type of timbre from the plurality of probability distributions specified by the timbre data Z for different timbres. The timbre data Z indicating the probability distribution for each timbre also corresponds to data indicating a timbre.

[0104] (4) In each of the above-described embodiments, the generation unit 53 generates the timbre data Z from the acoustic data F and the accompaniment data C, but the generation unit 53 may generate the timbre data Z from only the acoustic data F. In other words, the accompaniment data C may be omitted. As can be understood from the above explanation, the trained model 60 is comprehensively expressed as a model that has learned the relationship between the acoustics of the reference music piece (acoustic data Ft) and the timbre (timbre data Zt) used in the reference music piece.

[0105] (5) In each of the above-described embodiments, a deep neural network is exemplified as the trained model 60, but the trained model 60 is not limited to a deep neural network. For example, a statistical estimation model such as a Hidden Markov Model (HMM) or a Support Vector Machine (SVM) may be used as the trained model 60. Specifically, the trained model 60 is as follows.

[0106] (5-1)HMM An HMM is a statistical estimation model in which multiple latent states are interconnected. Each of the multiple latent states of the HMM represents one of multiple different types of timbres (i.e., timbre data Z). In each latent state, acoustic data F is generated that represents the characteristics related to the timbre represented by the latent state. As in each of the above-mentioned forms, the acoustic data F is data that represents a time series of frequency characteristics such as MFCC, MSLS, or constant Q transform. Moreover, the multiple latent states that make up the HMM correspond respectively to multiple periods (hereinafter referred to as "processing periods") obtained by dividing the acoustic signal V on the time axis. Each processing period is, for example, a period obtained by dividing the target music piece into units of a predetermined number (single or multiple) of bars.

[0107] The first acquisition unit 51 generates acoustic data F for each processing period of the acoustic signal V. The generation unit 53 inputs the time series of the multiple acoustic data F generated for different processing periods of the acoustic signal V to a trained model 60 configured by an HMM. The generation unit 53 estimates the time series of the most likely timbre data Z using the HMM under the condition that the time series of the multiple acoustic data F have been observed. That is, the timbre data Z is output from the HMM for each processing period of the acoustic signal V. A dynamic programming algorithm such as the Viterbi algorithm is used to estimate the time series of the timbre data Z.

[0108] The HMM is established by supervised machine learning (learning process Sc) using multiple training data T including timbre data Z. In the learning process Sc, the transition probability and output probability in each latent state are iteratively updated so that the most likely time series of timbre data Z is output for multiple time series of acoustic data F.

[0109] (5-2)SVM An SVM is prepared for each of all combinations of two timbres selected from a plurality of timbres. For the SVM corresponding to the combination of two timbres, a hyperplane in a multidimensional space is established by machine learning (learning process Sc). The hyperplane is a boundary surface that separates a space in which input data X corresponding to one of the two timbres is distributed and a space in which input data X corresponding to the other timbre is distributed. The trained model 60 is composed of multiple SVMs corresponding to different combinations of timbres (multi-class SVM).

[0110] The generation unit 53 inputs input data X including audio data F and accompaniment data C to each of the multiple SVMs. The SVM corresponding to each combination selects one of two types of timbres associated with the combination depending on in which of two spaces separated by a hyperplane the input data X exists. The selection of timbre is performed in the same manner in each of the multiple SVMs corresponding to different combinations. The generation unit 53 generates timbre data Z representing the timbre that is selected the greatest number of times by the multiple SVMs among the multiple types of timbres.

[0111] As can be understood from the above examples, regardless of the type of trained model 60, the generation unit 53 functions as an element that inputs input data X to the trained model 60 and outputs tone color data Z indicating a tone color adapted to the target song from the trained model 60.

[0112] (6) In each of the above-described embodiments, supervised machine learning using multiple training data T is exemplified as the learning process Sc, but the learned model 60 may be established by unsupervised machine learning that does not require training data T, or by reinforcement learning that maximizes reward. For example, the learning processing unit 72 sets the reward function to "+1" when the timbre data Zt output by the provisional model 65 for the input data Xt of each training data T matches the timbre data Z of the training data T, and sets the reward function to "-1" when the two do not match. The learning processing unit 72 establishes the learned model 60 by iteratively updating multiple variables of the provisional model 65 so that the sum of the reward functions set for the multiple training data T is maximized. In addition, an example of unsupervised machine learning is machine learning using known clustering.

[0113] (7) In each of the above-described embodiments, the machine learning system 40 establishes the trained model 60, but the functions of the machine learning system 40 (the training data acquisition unit 71 and the learning processing unit 72) may be incorporated in the information processing system 30. In addition, the functions of the machine learning system 40 (the training data acquisition unit 71 and the learning processing unit 72) may be incorporated in the electronic musical instrument 20 in the third embodiment, or the information device 80 in the fourth embodiment.

[0114] (8) In each of the above-mentioned embodiments, the trained model 60 that has trained the relationship between the input data X of the reference music piece and the timbre data Z of the reference music piece is used to generate the timbre data Z from the input data X corresponding to the audio signal V, but the configuration and method for generating the timbre data Z from the input data X are not limited to the above examples. For example, a reference table in which the timbre data Z is associated with each of a plurality of different input data X may be used for the generation of the timbre data Z by the generation unit 53. The reference table is a data table in which the correspondence between the input data X and the timbre data Z is registered, and is stored in, for example, the storage device 32. The generation unit 53 of the analysis processing unit 50 searches the reference table for input data X that matches or is close to the input data X generated from the audio signal V, and obtains the timbre data Z associated with the input data X from the reference table among the plurality of timbre data Z. With the above-mentioned configuration, it is possible to specify a timbre that is suitable for a new piece of music, for example, in the same way as in each of the above-mentioned embodiments.

[0115] (9) The functions exemplified in each of the above-mentioned embodiments (the analysis processing unit 50, the music estimation unit 56, and the timbre identification unit 57) are realized by the cooperation of one or more processors constituting the control device (21, 31, 71) and the programs stored in the storage device (22, 32, 72), as described above. The above-mentioned programs can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, a recording medium that stores the program in the distribution device corresponds to the above-mentioned non-transitory recording medium.

[0116] G: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0117] An information processing system according to one aspect (aspect 1) includes a first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music, and a generation unit that outputs timbre data indicating a timbre adapted to the target piece of music from a trained model that has learned the relationship between the acoustics of a reference piece of music and the timbre used in the reference piece of music by inputting input data including the acquired acoustic data into the trained model. According to the above aspect, by inputting input data including the acoustic data representing the acoustics of the target piece of music into the trained model, timbre data indicating a timbre adapted to the target piece of music is generated. Therefore, for example, a timbre adapted to a new piece of music can be identified.

[0118] The sound represented by the sound data is, for example, a musical tone produced from a sound source such as an instrument by playing a piece of music. "Performance" includes the operation of various musical instruments as well as singing by a singer. "Tone data" is data in any format that indicates a tone suitable for a target piece of music. For example, identification information for identifying one type of tone is assumed as the tone data. However, the tone data is not limited to data that explicitly specifies one type of tone. For example, data indicating the probability (likelihood) for each of multiple types of tone, or data indicating a probability distribution (e.g., mean and variance) for each of multiple tone colors is also included in the concept of "tone data".

[0119] A timbre that is "adapted" to a target piece of music means a timbre that musically corresponds to the target piece of music. For example, a timbre that corresponds to the melody of the target piece of music, a timbre that harmonizes with the musical expression of the target piece of music, or any other timbre that is suitable for the target piece of music is interpreted as a "timbre that is adapted to the target piece of music."

[0120] The "trained model" is a statistical estimation model established by supervised machine learning using, for example, multiple training data. Each of the multiple training data is, for example, a combination of training input data and training tone data (ground truth value).

[0121] An information processing system according to a specific example (aspect 2) of aspect 1 includes a second acquisition unit that acquires accompaniment data indicating an accompaniment pattern corresponding to the target music piece, and the trained model is a model that has learned the relationship between the acoustics of the reference music piece, the accompaniment pattern of the reference music piece, and the timbre used in the reference music piece, and the input data includes the acoustic data and the accompaniment data. In the above aspect, input data including acoustic data indicating the acoustics of the target music piece and accompaniment data indicating the accompaniment pattern corresponding to the target music piece is input to the trained model. Therefore, it is possible to identify a timbre suitable for a combination of the acoustics of the target music piece and the accompaniment pattern of the target music piece.

[0122] An "accompaniment pattern" is an audio signal that represents the accompaniment sound of a piece of music. For example, identification information for identifying one of a plurality of accompaniment patterns is exemplified as "accompaniment data." The accompaniment data may specify, for example, an accompaniment pattern that is automatically estimated by analyzing the acoustics of the target piece of music, or an accompaniment pattern that is specified by the user.

[0123] In a specific example (Aspect 3) of Aspect 2, the trained model includes a first model that generates first data representing acoustic features of the target song from the acoustic data, a second model that generates second data representing features of the accompaniment pattern from the accompaniment data, and a third model that generates the tone data from intermediate data including the first data and the second data.

[0124] The information processing system according to a specific example (Aspect 4) of Aspects 1 to 3 further includes a timbre identification unit that identifies timbre data indicating a timbre suitable for a target piece of music from reference data in which timbres suitable for each of a plurality of pieces of music are registered, when the timbre suitable for the target piece of music is registered, and the generation unit generates the timbre data using the trained model, when the timbre for the target piece of music is not registered in the reference data. In the above aspects, when an appropriate timbre for the target piece of music is registered in the reference data, timbre data indicating the registered timbre is generated. Therefore, while generating timbre data indicating an appropriate timbre for a registered piece of music, it is also possible to identify an appropriate timbre for an unregistered piece of music (e.g., a new piece of music created by a user).

[0125] An information processing system according to a specific example (aspect 5) of aspect 4 further includes a music estimation unit that estimates a plurality of candidate music pieces corresponding to a music piece played by a user from the plurality of music pieces registered in the reference data, and the timbre identification unit identifies the timbre data corresponding to the target music piece from the reference data when the user selects one of the plurality of candidate music pieces as the target music piece. In the above aspect, when the user selects one of the plurality of candidate music pieces as the target music piece, the timbre data corresponding to the target music piece is identified from the reference data. Therefore, appropriate timbre data can be identified for the music piece played by the user.

[0126] In a specific example (aspect 6) of aspect 5, when the user selects a piece of music other than the plurality of candidate pieces of music as the target piece of music, the generation unit generates the timbre data using the trained model. According to the above aspect, appropriate timbre data can be generated for an unregistered piece of music other than the plurality of candidate pieces of music.

[0127] In a specific example (Aspect 7) of Aspect 5 or Aspect 6, comparison data representing the content of each of the plurality of pieces of music is registered in the reference data, and the music estimation unit estimates the plurality of candidate pieces of music by comparing the comparison data registered for each of the plurality of pieces of music with data representing the performance by the user. According to the above aspect, the candidate pieces of music can be estimated using the comparison data registered in the reference data together with the timbre data.

[0128] In a specific example (Aspect 8) of any one of Aspects 1 to 7, the acoustic data is data representing a time series of frequency characteristics of the acoustics of the target song. Examples of the frequency characteristics of the acoustics of the target song include an intensity spectrum such as an amplitude spectrum or a power spectrum, as well as Mel-Frequency Cepstrum Coefficient (MFCC), Mel-Scale Log Spectrum (MSLS), Constant-Q Transform (CQT), and the like.

[0129] An electronic musical instrument according to one aspect (aspect 9) of the present disclosure includes a performance receiving unit that receives a performance by a user, a first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music, a generation unit that outputs timbre data representing a timbre adapted to the target piece of music from a trained model that has learned the relationship between the acoustics of a reference piece of music and timbres used in the reference piece of music by inputting input data including the acquired acoustic data into the trained model, and a reproduction unit that reproduces a timbre of a timbre corresponding to the timbre data in accordance with the performance received by the performance receiving unit. The "performance received by the performance receiving unit" is, for example, a performance on multiple keys.

[0130] In a specific example (aspect 10) of aspect 9, the performance receiving unit includes a keyboard on which a plurality of keys corresponding to different pitches are arranged, and the playback unit plays back a sound having a pitch corresponding to a key played by the user among the plurality of keys and a tone corresponding to the tone data. In other words, the electronic musical instrument is a keyboard instrument equipped with a keyboard. According to the above aspect, it is possible to play back a sound having a pitch played by the user on the keyboard and a tone corresponding to the tone data.

[0131] An information processing method according to one aspect (Aspect 11) of the present disclosure acquires acoustic data representing the acoustics of a target piece of music, and inputs input data including the acquired acoustic data into a trained model that has learned the relationship between the acoustics of a reference piece of music and the timbres used in the reference piece of music, thereby outputting timbre data indicating a timbre adapted to the target piece of music from the trained model. The aforementioned aspects (Aspects 2 to 8) that can be employed for the information processing system can also be employed for the information processing method according to Aspect 11.

[0132] A program according to one embodiment of the present disclosure causes a computer system to function as a first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music, and a generation unit that outputs tone data indicating a tone adapted to the target piece of music from a trained model that has learned the relationship between the acoustics of a reference piece of music and the tone used in the reference piece of music by inputting input data including the acquired acoustic data into the trained model. [Explanation of symbols]

[0133] 100...performance system, 10...signal supply device, 20...electronic musical instrument, 21,31,41,81...control device, 22,32,42,82...storage device, 23,33,43...communication device, 24...performance device, 25...operation device, 26...display device, 27...sound source device, 28...sound emission device, 29...playback unit, 30...information processing system, 40...machine learning system, 50...analysis processing unit, 51...first acquisition unit, 52...second acquisition unit, 53...generation unit, 56...music estimation unit, 57...timbre identification unit, 60...trained model, 61...first model, 62...second model, 63...third model, 65...tentative model, 71...training data acquisition unit, 72...learning processing unit, 80...information device.

Claims

1. Acquire audio data representing the audio of a target piece of music; By processing input data including the acquired acoustic data, timbre data indicating a timbre suited to the target piece of music is generated. An information processing method implemented by a computer system.

2. Furthermore, acquiring accompaniment data indicating an accompaniment pattern corresponding to the target piece of music; The input data includes the acquired acoustic data and the acquired accompaniment data.

2. The information processing method according to claim 1.

3. Furthermore, When a timbre suitable for the target piece of music is registered in reference data in which timbres suitable for each of a plurality of pieces of music are registered, timbre data indicating the timbre is identified from the reference data; In the generation of the timbre data, if the timbre of the target piece of music is not registered in the reference data, the timbre data is generated.

2. The information processing method according to claim 1.

4. Furthermore, Estimating a plurality of candidate pieces of music corresponding to the piece of music played by the user from the plurality of pieces of music registered in the reference data; Further comprising: In the case where the user selects any one of the plurality of candidate pieces of music as the target piece of music, the timbre data corresponding to the target piece of music is identified from the reference data. The information processing method according to claim 3.

5. In generating the tone color data, when the user selects a song other than the plurality of candidate songs as the target song, the tone color data is generated. The information processing method according to claim 4.

6. The reference data includes comparison data for each of the plurality of songs that represents the content of the song, In estimating the plurality of candidate songs, the comparison data registered for each of the plurality of songs is compared with data representing the performance by the user to estimate the plurality of candidate songs. The information processing method according to claim 4 or 5.

7. The acoustic data is data representing a time series of frequency characteristics related to the acoustics of the target song.

2. The information processing method according to claim 1.

8. A first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music; a generation unit that processes input data including the acquired acoustic data to generate timbre data indicating a timbre suited to the target piece of music; An information processing system comprising:

9. A performance reception unit that receives performances by users; a first acquisition unit that acquires audio data representing the audio of a target piece of music; a generation unit that processes input data including the acquired acoustic data to generate timbre data indicating a timbre suited to the target piece of music; a playback unit that plays back a sound of a tone color corresponding to the tone color data in response to the performance received by the performance reception unit; An electronic musical instrument comprising:

10. A first acquisition unit that acquires acoustic data representing the acoustics of a target piece of music; and a generation unit that processes input data including the acquired acoustic data to generate timbre data indicating a timbre suited to the target piece of music; A program that causes a computer system to function as a

Images