Information processing system, electronic musical instrument, method of processing information, and machine learning system

Abstract

To achieve effective performance practice according to a user's performance tendency.SOLUTION: An information processing system includes a receiving unit that receives performance data representing a performance of a piece of music by a practitioner, and a generating unit that uses the performance data to generate tendency data representing a performance tendency designated by an instructor for the performance.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a technique for assisting in playing musical instruments such as electronic musical instruments. [Background technology]

[0002] Various techniques for supporting the performance of musical instruments such as electronic musical instruments have been proposed. For example, Patent Document 1 discloses a technique for calculating statistical values ​​such as standard deviation from the difference between parameters of music data prepared in advance and parameters of performance data representing a performance by a user, and tallying up the statistical values ​​by a method according to the type of the parameters. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-55635 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, simply presenting a user with an evaluation value that is the result of evaluating the performance makes it difficult to effectively practice a performance based on the performance tendencies of each individual user (e.g., tendencies to make performance mistakes, etc.) In consideration of the above circumstances, one aspect of the present disclosure has an object to realize effective performance practice in accordance with the performance tendencies of the user. [Means for solving the problem]

[0005] In order to solve the above problems, an information processing system according to one aspect of the present disclosure includes a performance data acquisition unit that acquires performance data representing a musical piece played by a user, a tendency identification unit that generates tendency data representing a performance tendency of the user by inputting the performance data acquired by the performance data acquisition unit into a first learned model that has learned the relationship between learning performance data representing the musical piece performance and learning tendency data that represents the performance tendency represented by the learning performance data, and a practice phrase identification unit that identifies a practice phrase according to the tendency data generated by the tendency identification unit.

[0006] An electronic musical instrument according to one aspect of the present disclosure includes a performance receiving unit that receives a musical piece performance by a user, a performance data acquisition unit that acquires performance data representing the performance received by the performance receiving unit, a tendency identification unit that outputs tendency data representing the performance tendency of the user from the first learned model by inputting the performance data acquired by the performance data acquisition unit into a first learned model that has learned the relationship between learning performance data representing the musical piece performance and learning tendency data that represents the performance tendency represented by the learning performance data, a practice phrase identification unit that uses the tendency data output by the tendency identification unit to identify practice phrases that correspond to the playing tendency of the user, and a presentation processing unit that presents the practice phrases to the user.

[0007] An information processing method according to one aspect of the present disclosure acquires performance data representing a musical piece played by a user, and inputs the acquired performance data into a first trained model that has learned the relationship between training performance data representing the musical piece performance and training tendency data representing the performance tendency represented by the training performance data, thereby generating tendency data representing the performance tendency of the user, and identifying practice phrases corresponding to the tendency data.

[0008] A machine learning system according to one embodiment of the present disclosure includes a first learning data acquisition unit that acquires performance data representing a performance of a musical piece by a user and comment data representing a time point within the musical piece and a tendency of the performance at that time point, and a first learning processing unit that establishes a first learned model that learns a relationship between the learning performance data and the learning tendency data by machine learning using first learning data that represents a combination of learning performance data representing a performance within a section of the performance data that includes a time point represented by the comment data and learning tendency data that represents the tendency of the performance represented by the comment data. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating an example of the configuration of a performance system in a first embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating a configuration of an electronic musical instrument. [Diagram 3] FIG. 1 is a block diagram illustrating a configuration of an information processing system. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing system. [Diagram 5] 11 is a flowchart illustrating a specific procedure of a specification process. [Figure 6] FIG. 1 is a block diagram illustrating a configuration of a machine learning system. [Figure 7] FIG. 1 is a block diagram illustrating an example of the functional configuration of a machine learning system. [Figure 8] FIG. 13 is a block diagram illustrating a configuration of an information device used by an instructor. [Figure 9] FIG. 13 is a schematic diagram of indicated data. [Figure 10] 11 is a flowchart illustrating a specific procedure of a preparation process. [Figure 11] 11 is a flowchart illustrating a specific procedure of a learning process. [Figure 12] FIG. 11 is a block diagram illustrating a functional configuration of an information processing system according to a second embodiment. [Figure 13]10 is a flowchart illustrating a procedure of a specification process in the second embodiment. [Figure 14] FIG. 13 is a block diagram illustrating an example of a functional configuration of an information processing system according to a third embodiment. [Figure 15] 13 is a flowchart illustrating a procedure of a specification process in the third embodiment. [Figure 16] FIG. 13 is a block diagram illustrating the functional configuration of a machine learning system in a third embodiment. [Figure 17] 13 is a flowchart illustrating a procedure of a learning process in the third embodiment. [Figure 18] FIG. 13 is a block diagram illustrating an example of the functional configuration of an electronic musical instrument according to a fourth embodiment. [Figure 19] FIG. 13 is a block diagram illustrating a functional configuration of an information device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] A: First embodiment 1 is a block diagram illustrating the configuration of a performance system 100 according to the first embodiment. The performance system 100 is a computer system that allows a user U of an electronic musical instrument 10 to practice playing the electronic musical instrument 10, and includes the electronic musical instrument 10, an information processing system 20, and a machine learning system 30. The components that make up the performance system 100 communicate with each other via a communication network 200 such as the Internet. Note that while the performance system 100 actually includes multiple electronic musical instruments 10, the following description focuses on one arbitrary electronic musical instrument 10 for convenience.

[0011] 2 is a block diagram illustrating the configuration of an electronic musical instrument 10. The electronic musical instrument 10 is a performance device used by a user U to play a musical piece. The electronic musical instrument 10 of the first embodiment is an electronic keyboard instrument having a plurality of keys operated by the user U. The electronic musical instrument 10 is realized by a computer system including a control device 11, a storage device 12, a communication device 13, a performance device 14, a display device 15, a sound source device 16, and a sound emission device 17. The electronic musical instrument 10 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0012] The control device 11 is composed of one or more processors that control each element of the electronic musical instrument 10. For example, the control device 11 is composed of one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0013] The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that the storage device 12 may be a portable recording medium that is detachable from the electronic musical instrument 10, or a recording medium (e.g., cloud storage) to which the control device 11 can write or read via the communication network 200.

[0014] The storage device 12 of the first embodiment stores a plurality of pieces of music data X representing different pieces of music. The music data X of each piece of music specifies a time sequence of a plurality of notes that make up a part or all of the piece of music. Specifically, the music data X specifies the pitch and sound duration of each note in the piece of music. The music data X is data in a format that complies with, for example, the Musical Instrument Digital Interface (MIDI) standard.

[0015] The communication device 13 communicates with the information processing system 20 via the communication network 200. The communication between the communication device 13 and the communication network 200 may be either wired communication or wireless communication. Also, a communication device 13 separate from the electronic musical instrument 10 may be connected to the electronic musical instrument 10 by wire or wirelessly. An example of the communication device 13 separate from the electronic musical instrument 10 is an information terminal such as a smartphone or a tablet terminal.

[0016] The display device 15 displays an image under the control of the control device 11. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel are used as the display device 15. The display device 15 displays the musical score of a piece of music performed by a user U, for example, by using music data X of the piece of music.

[0017] The performance device 14 is an input device that accepts a performance by the user U. Specifically, the performance device 14 has a keyboard on which a plurality of keys corresponding to different pitches are arranged. The user U plays a piece of music by sequentially operating the desired keys on the performance device 14. The performance device 14 is an example of a "performance accepting unit."

[0018] The control device 11 generates performance data Y representing the performance of a musical piece by the user U. Specifically, the performance data Y specifies the pitch and the duration of each of a plurality of notes specified by the user U through an operation on the performance device 14. The performance data Y is time-series data in a format conforming to the MIDI standard, for example, similar to the musical piece data X. The communication device 13 transmits the performance data Y representing the performance of a musical piece by the user U and the musical piece data X of the musical piece to the information processing system 20. The musical piece data X is data representing an exemplary or standard performance of a musical piece, and the performance data Y is data representing the actual performance of the musical piece by the user U. Therefore, the notes specified by the musical piece data X and the notes specified by the performance data Y are mutually correlated but do not completely match. The difference between the musical piece data X and the performance data Y is particularly noticeable in parts of the musical piece where the user U is likely to make performance mistakes or where the user U is not good at playing.

[0019] The sound source device 16 generates an audio signal A according to a performance on the performance device 14. The audio signal A is a signal representing a waveform of a musical tone instructed by a performance on the performance device 14. Specifically, the sound source device 16 is a MIDI sound source that generates an audio signal A representing a musical tone of each note specified in a time series by the performance data Y. That is, the sound source device 16 generates an audio signal A representing a musical tone of a pitch corresponding to a key pressed by the user U among a plurality of keys on the performance device 14. Note that the control device 11 may realize the function of the sound source device 16 by executing a program stored in the storage device 12. That is, the sound source device 16 dedicated to generating the audio signal A is omitted.

[0020] The sound emitting device 17 emits the performance sound represented by the audio signal A. For example, a speaker or a headphone is used as the sound emitting device 17. As can be understood from the above description, the sound source device 16 and the sound emitting device 17 in the first embodiment function as a reproduction system 18 that reproduces musical sounds according to the performance by the user U.

[0021] 3 is a block diagram illustrating an example of the configuration of an information processing system 20. The information processing system 20 provides a user U with musical phrases (hereinafter referred to as "practice phrases") Z suitable for practicing performance by the user U. The information processing system 20 is realized by a computer system including a control device 21, a storage device 22, and a communication device 23. The information processing system 20 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0022] The control device 21 is composed of one or more processors that control each element of the information processing system 20. For example, the control device 21 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The communication device 23 communicates with each of the electronic musical instrument 10 and the machine learning system 30 via the communication network 200. Note that the communication between the communication device 23 and the communication network 200 may be either wired communication or wireless communication.

[0023] The storage device 22 is a single or multiple memories that store programs executed by the control device 21 and various data used by the control device 21. The storage device 22 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that a portable recording medium that is detachable from the information processing system 20, or a recording medium (e.g., cloud storage) to which the control device 21 can write or read via the communication network 200, may be used as the storage device 22.

[0024] 4 is a block diagram illustrating a functional configuration of the information processing system 20. The storage device 22 stores a plurality of practice phrases Z corresponding to different trend data D. In other words, a table in which each of the plurality of trend data D and each of the plurality of practice phrases Z are associated with each other is stored in the storage device 22.

[0025] The tendency data D is data in any format that indicates the tendency of a performer to perform (hereinafter referred to as "performance tendency"). The performance tendency is, for example, the tendency of a performer to make performance mistakes or the tendency of a performer to be weak at a performance method. For example, the tendency data D specifies one of a plurality of performance tendencies, such as "delaying key pressing timing," "pressing other keys adjacent to the intended key," "playing the wrong pitch," "weakness in jump progression," "weakness in playing chords," and "weakness in finger underpassing." A jump progression is a portion in which two notes with a pitch difference exceeding a predetermined value (e.g., a third) are played one after the other. A finger underpassing is a performance method in which the fingers of one hand move to pass under the finger pressing the key corresponding to one note to play a higher note.

[0026] A practice phrase Z is time-series data representing a piece of music composed of multiple notes, and specifically, a melody suitable for practicing the electronic musical instrument 10 (e.g., a part or all of a practice piece). A practice phrase Z is composed of a time series of single notes or chords. A practice phrase Z corresponding to each tendency data D represents a piece of music suitable for improving the performance tendency specified by the tendency data D. For example, for tendency data D of a performance tendency "poor at jump progressions," a practice phrase Z rich in jump progressions is registered. Also, for tendency data D of a performance tendency "poor at playing chords," a practice phrase Z rich in chords is registered. A practice phrase Z is, for example, data in MIDI format that specifies the pitch and sound duration for each of multiple notes.

[0027] The control device 21 of the information processing system 20 executes a program stored in the storage device 22 to realize multiple elements (a performance data acquisition unit 71, a tendency identification unit 72, and a practice phrase identification unit 73) for identifying a practice phrase Z from music piece data X and performance data Y.

[0028] The performance data acquisition unit 71 acquires performance data Y representing a musical piece performed by the user U. Specifically, the performance data acquisition unit 71 receives the musical piece data X and the performance data Y transmitted from the electronic musical instrument 10 via the communication device 23. The performance data acquisition unit 71 generates control data C including the musical piece data X and the performance data Y.

[0029] The tendency identification unit 72 generates tendency data D indicating the playing tendency of the user U in accordance with the control data C. The tendency identification unit 72 uses the trained model Ma to generate the tendency data D. The trained model Ma is an example of a “first trained model.”

[0030] There is a correlation between the difference or similarity between the score (music data X) of a piece of music played by a performer and the actual performance (performance data Y) by the performer, and the performance tendency (tendency data D) of the performer. For example, when the time of sounding of each note differs between the music data X and the performance data Y, a performance tendency of "the time of pressing the key is different" is estimated. When the performance data Y specifies another note close to the note represented by the music data X, a performance tendency of "pressing another key adjacent to the target key" is estimated. When the difference between the music data X and the performance data Y is significant at a point in the piece of music where a jump progression exists, a performance tendency of "not good at jump progression" is estimated. The trained model Ma is a statistical estimation model that has learned the above-mentioned tendencies. In other words, the trained model Ma is a statistical estimation model that has learned the relationship between the combination of the music data X and the performance data Y (i.e., the control data C) and the tendency data D that represents the performance tendency of the performer. The tendency identification unit 72 inputs the control data C including the music piece data X and the performance data Y to the trained model Ma, and outputs tendency data D representing the performance tendency of the user U from the trained model Ma.

[0031] The trained model Ma is composed of, for example, a deep neural network (DNN). For example, any type of neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN), is used as the trained model Ma. The trained model Ma may be composed of a combination of multiple types of deep neural networks. In addition, additional elements such as a long short-term memory (LSTM) may be installed in the trained model Ma.

[0032] The trained model Ma is realized by a combination of a program that causes the control device 21 to execute a calculation to generate trend data D from control data C, and a plurality of variables (specifically, weights and biases) that are applied to the calculation. The program and the plurality of variables that realize the trained model Ma are stored in the storage device 22. The numerical values ​​of each of the plurality of variables that define the trained model Ma are set in advance by machine learning.

[0033] The practice phrase identification unit 73 uses the tendency data D identified by the tendency identification unit 72 to identify a practice phrase Z corresponding to the performance tendency of the user U. Specifically, the practice phrase identification unit 73 searches the storage device 22 for a practice phrase Z corresponding to the tendency data D identified by the tendency identification unit 72, among a plurality of practice phrases Z stored in the storage device 22. That is, a practice phrase Z suitable for improving the performance tendency of the user U represented by the tendency data D is identified.

[0034] The practice phrase Z identified by the practice phrase identification unit 73 is transmitted from the communication device 23 to the electronic musical instrument 10. The communication device 13 of the electronic musical instrument 10 receives the practice phrase Z transmitted from the information processing system 20. The control device 11 causes the display device 15 to display the musical score of the practice phrase Z. The user U plays the practice phrase Z while checking the musical score displayed on the display device 15.

[0035] FIG. 5 is a flowchart illustrating a specific procedure of a process (hereinafter, referred to as a "specific process") Sa executed by the control device 21 of the information processing system 20. As shown in FIG.

[0036] When the identification process Sa is started, the performance data acquisition unit 71 waits until the music piece data X and the performance data Y transmitted from the electronic musical instrument 10 are received by the communication device 23 (Sa1: NO). When the performance data acquisition unit 71 acquires the music piece data X and the performance data Y (Sa1: YES), the tendency identification unit 72 inputs the control data C including the music piece data X and the performance data Y into the trained model Ma, thereby outputting tendency data D from the trained model Ma (Sa2). The practice phrase identification unit 73 identifies a practice phrase Z corresponding to the tendency data D from among the multiple practice phrases Z stored in the storage device 22 (Sa3). The practice phrase identification unit 73 transmits the practice phrase Z from the communication device 23 to the electronic musical instrument 10 (Sa4).

[0037] As described above, in the first embodiment, performance data Y representing a performance of a piece of music by a user U is input to a trained model Ma to generate tendency data D representing the performance tendency of the user U, and a practice phrase Z is identified according to the tendency data D. Therefore, when the user U plays the practice phrase Z, effective practice according to the performance tendency of the user U is realized.

[0038] In the first embodiment, the practice phrase Z corresponding to the performance tendency of the user U is identified from among a plurality of practice phrases Z corresponding to different performance tendencies (tendency data D). Therefore, the processing load of identifying the practice phrase Z according to the performance tendency of the user U is reduced.

[0039] The machine learning system 30 in Fig. 1 generates the trained model Ma exemplified above. Fig. 6 is a block diagram illustrating the configuration of the machine learning system 30. The machine learning system 30 includes a control device 31, a storage device 32, and a communication device 33. Note that the machine learning system 30 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0040] The control device 31 is composed of one or more processors that control each element of the machine learning system 30. For example, the control device 31 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The communication device 33 communicates with the information processing system 20 via the communication network 200. Note that the communication between the communication device 33 and the communication network 200 may be either wired communication or wireless communication.

[0041] The storage device 32 is a single or multiple memories that store the programs executed by the control device 31 and various data used by the control device 31. The storage device 32 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the machine learning system 30, or a recording medium (e.g., cloud storage) that can be written to or read by the control device 31 via the communication network 200 may be used as the storage device 32.

[0042] 7 is a block diagram illustrating an example of a functional configuration of the machine learning system 30. The control device 31 executes a program stored in the storage device 32 to function as a plurality of elements (a learning data acquisition unit 81a and a learning processing unit 82a) for establishing a trained model Ma by machine learning.

[0043] The learning processing unit 82a establishes a learned model Ma by supervised machine learning (learning processing Sc described later) using a plurality of learning data Ta. The learning data acquisition unit 81a acquires a plurality of learning data Ta. The plurality of learning data Ta acquired by the learning data acquisition unit 81a is stored in the storage device 32. Each of the plurality of learning data Ta is composed of a combination of learning control data Ct and learning tendency data Dt. The control data Ct includes learning music data Xt and learning performance data Yt. The music data Xt is an example of "learning music data", the performance data Yt is an example of "learning performance data", and the tendency data Dt is an example of "learning tendency data". Moreover, the music represented by the music data Xt is an example of "reference music". The learning data acquisition unit 81a is an example of a "first learning data acquisition unit", and the learning processing unit 82a is an example of a "first learning processing unit". Moreover, the learning data Ta is an example of "first learning data".

[0044] As illustrated in FIG. 7, the learning data Ta is generated using the results of a performance of a musical piece by a learner U1 and instruction on the performance by an instructor U2. The learner U1 plays the musical piece using an electronic musical instrument 10. The instructor U2 evaluates and provides instruction on the performance of the learner U1 using an information device 40. The information device 40 is, for example, an information terminal such as a smartphone or a tablet terminal. The learner U1 and the instructor U2 are located, for example, in remote locations. However, the learner U1 and the instructor U2 may be located in the same place.

[0045] The electronic musical instrument 10 transmits music piece data X0 representing a piece of music and performance data Y0 representing a performance of the piece of music by a learner U1 to the information device 40 and the machine learning system 30. The music piece data X0 specifies a time series of multiple notes that make up the piece of music, similar to the above-mentioned music piece data X. The performance data Y0 specifies a time series of multiple notes instructed by the learner U1 by operating the performance device 14, similar to the above-mentioned performance data Y.

[0046] 8 is a block diagram illustrating the configuration of an information device 40. The information device 40 is a computer system that allows an instructor U2 to evaluate and teach a performance of an electronic musical instrument 10 by a learner U1, and includes a control device 41, a storage device 42, a communication device 43, an operation device 44, a display device 45, and a playback system 46. The information device 40 may be realized as a single device, or may be realized as a plurality of devices configured separately from each other.

[0047] The control device 41 is composed of one or more processors that control each element of the information device 40. For example, the control device 41 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC.

[0048] The storage device 42 is a single or multiple memories that store the programs executed by the control device 41 and various data used by the control device 41. The storage device 42 is configured with a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that a portable recording medium that is detachable from the information device 40, or a recording medium (e.g., cloud storage) to which the control device 41 can write or read via the communication network 200 may be used as the storage device 42.

[0049] The communication device 43 communicates with each of the electronic musical instrument 10 and the machine learning system 30 via the communication network 200. Note that the communication between the communication device 43 and the communication network 200 may be either wired communication or wireless communication. The communication device 43 receives, for example, the music piece data X0 and the performance data Y0 transmitted from the electronic musical instrument 10.

[0050] The operation device 44 is an input device that receives instructions from the instructor U2. The operation device 44 is, for example, a plurality of operators operated by the instructor U2, or a touch panel that detects contact by the instructor U2. The display device 45 displays an image under the control of the control device 41. Specifically, the display device 45 displays a time series of notes specified by the performance data Y received by the communication device 43. That is, an image representing the performance by the learner U1 is displayed on the display device 45. Note that the time series of notes specified by the music data X may be displayed in parallel with the notes of the performance data Y. The reproduction system 46 reproduces the musical tones of each note specified by the performance data Y, similar to the reproduction system 18 of the electronic musical instrument 10. That is, the musical tones played by the learner U1 are reproduced by the reproduction system 46.

[0051] The instructor U2 can check the performance of the piece of music by the learner U1 by listening to the sound reproduced by the reproduction system 46 while viewing the image displayed by the display device 45. The instructor U2 operates the operation device 44 to input the performance tendency to be pointed out in the performance of the piece of music by the learner U1. The instructor U2 specifies the performance tendency of the performance of the piece of music by the learner U1 and the time point at which the performance tendency is observed in the piece of music. The instructor U2 selects the performance tendency from a plurality of options by operating the operation device 44, for example. For example, one of a plurality of performance tendencies such as "the timing of pressing the key is off," "pressing a key adjacent to the intended key," "making a mistake in pitch," "poor jump progression," "poor performance of chords," and "poor performance of short notes such as sixteenth notes quickly" is selected as a point of criticism regarding the performance of the learner U1.

[0052] The control device 41 generates the instruction data P in response to an instruction from the instructor U2. FIG. 9 is a schematic diagram of the instruction data P. The instruction data P includes tendency data Dt and time data τ for each instruction from the instructor U2. The tendency data Dt is data representing the performance tendency pointed out by the instructor U2. The time data τ is data representing the time at which the performance tendency is observed in the music piece. As can be understood from the above explanation, the instruction data P is data representing a time in the music piece and the performance tendency at that time.

[0053] The communication device 43 transmits the instruction data P generated by the control device 41 to the electronic musical instrument 10 and the machine learning system 30. The communication device 13 of the electronic musical instrument 10 receives the instruction data P transmitted from the information device 40. The control device 11 displays the performance tendency represented by the instruction data P on the display device 15. The learner U1 can confirm the instruction (performance tendency) from the instructor U2 by visually checking the image on the display device 15.

[0054] 7, the learning data acquisition unit 81a in the machine learning system 30 receives, via the communication device 33, the music piece data X0 and performance data Y0 transmitted from the electronic musical instrument 10, and the instruction data P transmitted from the information device 40. The learning data acquisition unit 81a generates learning data Ta using the music piece data X0, performance data Y0, and instruction data P. Note that the electronic musical instrument 10 is an example of a "first device," and the information device 40 is an example of a "second device."

[0055] 10 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as "preparatory process") Sb in which the learning data acquisition unit 81a generates learning data Ta. For example, the preparation process Sb is started when the communication device 33 receives music data X0, performance data Y0, and instruction data P. When the preparation process Sb is started, the learning data acquisition unit 81a acquires the music data X0, performance data Y0, and instruction data P from the communication device 33 (Sb1).

[0056] The learning data acquisition unit 81a extracts, as the music piece data Xt, a portion of the music piece data X0 within a section (hereinafter referred to as a "specific section") including a time point specified by the time data τ of the instruction data P (Sb2). The specific section is, for example, a section of a predetermined length with the time point specified by the time data τ as its midpoint. The learning data acquisition unit 81a also extracts, as the performance data Yt, a portion of the performance data Y0 within the specific section including a time point specified by the time data τ of the instruction data P (Sb3). That is, for each of the music piece data X0 and the performance data Y0, a specific section including a time point at which the instructor U2 pointed out the performance tendency is extracted.

[0057] The learning data acquisition unit 81a generates learning control data Ct including the music piece data Xt and performance data Yt generated by the above procedure (Sb4).Then, the learning data acquisition unit 81a generates learning data Ta by making the learning control data Ct correspond to the tendency data Dt included in the indication data P (Sb5).

[0058] By repeating the preparation process Sb illustrated above, a large number of learning data Ta are generated for a variety of musical pieces played by a large number of learners U1, which include musical piece data Xt and performance data Yt corresponding to specific sections, and tendency data Dt of the performance tendency pointed out by the instructor U2 for the specific sections.

[0059] 11 is a flowchart illustrating a specific procedure of a learning process Sc in which the control device 31 of the machine learning system 30 establishes a trained model Ma. The learning process Sc is also expressed as a method of generating a trained model Ma by machine learning (a method of generating a trained model).

[0060] When the learning process Sc is started, the learning processing unit 82a selects one of the multiple pieces of learning data Ta (hereinafter referred to as "selected learning data Ta") stored in the storage device 32 (Sc1). As illustrated in Fig. 7, the learning processing unit 82a inputs the control data Ct of the selected learning data Ta to an initial or provisional model (hereinafter referred to as "provisional model Ma0") (Sc2), and obtains the tendency data D output by the provisional model Ma0 in response to the input (Sc3).

[0061] The learning processing unit 82a calculates a loss function that represents the error between the trend data D generated by the provisional model Ma0 and the trend data Dt of the selected learning data Ta (Sc4). The learning processing unit 82a updates multiple variables of the provisional model Ma0 so that the loss function is reduced (ideally minimized) (Sc5). For example, the backpropagation method is used to update the multiple variables according to the loss function.

[0062] The learning processing unit 82a determines whether a predetermined termination condition is satisfied (Sc6). The termination condition is, for example, that the loss function falls below a predetermined threshold or that the change amount of the loss function falls below a predetermined threshold. If the termination condition is not satisfied (Sc6: NO), the learning processing unit 82a selects the unselected learning data Ta as new selected learning data Ta (Sc1). That is, the process of updating a plurality of variables of the provisional model Ma0 (Sc2-Sc5) is repeated until the termination condition is satisfied (Sc6: YES). If the termination condition is satisfied (Sc6: YES), the learning processing unit 82a terminates the update (Sc2-Sc5) of a plurality of variables that define the provisional model Ma0. The provisional model Ma0 at the time when the termination condition is satisfied is determined as the trained model Ma. That is, the plurality of variables of the trained model Ma are determined to be the numerical values ​​at the time when the learning process Sc is terminated.

[0063] As can be understood from the above explanation, the trained model Ma outputs statistically valid trend data D for unknown control data C, based on the underlying relationship between the control data Ct and trend data Dt in multiple pieces of training data Ta. In other words, the trained model Ma is a statistical learning model that has learned the relationship between a performer's performance of a piece of music (control data C) and the performance trend of the performer (tendency data D), as described above.

[0064] The learning processing unit 82a transmits the trained model Ma established by the above procedure to the information processing system 20 from the communication device 33 (Sc7). Specifically, the learning processing unit 82a transmits multiple variables of the trained model Ma from the communication device 33 to the information processing system 20. The control device 21 of the information processing system 20 stores the trained model Ma received from the machine learning system 30 in the storage device 22. Specifically, multiple variables that define the trained model Ma are stored in the storage device 22.

[0065] B: Second embodiment A second embodiment will be described. Note that, for elements having the same functions as those in the first embodiment in each of the following exemplary aspects, the same reference numerals as those in the first embodiment will be used, and detailed descriptions of each will be omitted as appropriate.

[0066] 12 is a block diagram illustrating a functional configuration of an information processing system 20 in the second embodiment. In the first embodiment, a plurality of practice phrases Z are stored in the storage device 22. In the second embodiment, one reference phrase Zref is stored in the storage device 22 instead of the plurality of practice phrases Z in the first embodiment.

[0067] Like the practice phrase Z in the first embodiment, the reference phrase Zref is time-series data representing a piece of music composed of a plurality of notes. Specifically, the reference phrase Zref is a melody (e.g., a part or all of a practice piece) suitable for practicing the electronic musical instrument 10. The practice phrase identification unit 73 in the second embodiment generates the practice phrase Z by editing the reference phrase Zref in accordance with the tendency data D generated by the tendency identification unit 72. Specifically, the practice phrase identification unit 73 edits the reference phrase Zref so as to reduce the difficulty of performance for a portion of the reference phrase Zref related to the performance tendency specified by the tendency data D.

[0068] 13 is a flowchart illustrating a specific procedure of the specific process Sa in the second embodiment. The specific process Sa in the second embodiment is a process in which step Sa3 in the specific process Sa in the first embodiment is replaced with step Sa13.

[0069] Acquisition of music piece data X and performance data Y by the performance data acquisition unit 71 (Sa1), and generation of trend data D by the trend identification unit 72 (Sa2) are the same as in the first embodiment. A practice phrase identification unit 73 in the second embodiment generates practice phrase Z by editing the reference phrase Zref stored in the storage device 22 in accordance with the trend data D (Sa13). The process (Sa4) in which the practice phrase identification unit 73 transmits practice phrase Z to the electronic musical instrument 10 is the same as in the first embodiment. A specific example of editing the reference phrase Zref (Sa13) will be described below.

[0070] For example, if the tendency data D indicates a performance tendency of "difficulty in playing chords," the practice phrase identification unit 73 generates the practice phrase Z by changing one or more chords included in the reference phrase Zref. For example, for a chord that includes more than a predetermined number of constituent tones, the practice phrase identification unit 73 omits one or more constituent tones, for example, other than the root note, from among the multiple constituent tones. Also, for a chord in which the pitch difference between the lowest note and the highest note exceeds a predetermined value, a predetermined number of constituent tones including the highest note are omitted. Omission of constituent tones reduces the difficulty of playing the chord. As shown in the above example, the editing of the reference phrase Zref by the practice phrase identification unit 73 includes changing the chord.

[0071] Furthermore, if the tendency data D indicates a performance tendency of "not good at jump progressions," the practice phrase identification unit 73 generates the practice phrase Z by omitting or changing the jump progression included in the reference phrase Zref. For example, the practice phrase identification unit 73 omits the latter note of the two notes involved in the jump progression. Furthermore, the practice phrase identification unit 73 changes the latter note of the two notes involved in the jump progression to another note on the lower pitch side. As shown in the above examples, the editing of the reference phrase Zref by the practice phrase identification unit 73 includes omitting or changing the jump progression.

[0072] The reference phrase Zref includes, for example, a specification of a playing method such as fingering. Specifically, the practice phrase Z includes, for each of a plurality of notes, a specification of the number of the finger that should play the corresponding note. When the tendency data D indicates a playing tendency of "poor finger undercutting", the practice phrase identification unit 73 generates the practice phrase Z by changing the fingering for the reference phrase Zref. For example, assuming that pressing a key with the little finger is difficult for a beginner performer, the practice phrase identification unit 73 changes the number of the note in the reference phrase Zref for which the little finger number is specified to the number of the other finger than the little finger. In the electronic musical instrument 10 that receives the edited practice phrase Z, the fingering (the finger number for each note) changed by the practice phrase identification unit 73 is displayed on the display device 15 together with the score of the practice phrase Z. As shown in the above example, the editing of the reference phrase Zref by the practice phrase identification unit 73 includes a change in the playing method of the musical instrument.

[0073] The second embodiment achieves the same effects as the first embodiment. In addition, in the second embodiment, the practice phrase Z is generated by editing the reference phrase Zref, so that the user U can be provided with an appropriate practice phrase Z according to the level of the performance technique of the user U.

[0074] C: Third embodiment 14 is a block diagram illustrating a functional configuration of an information processing system 20 in the third embodiment. In the first embodiment, a configuration was exemplified in which the practice phrase identification unit 73 identifies a practice phrase Z corresponding to the tendency data D of a user U from among a plurality of practice phrases Z stored in the storage device 22. The practice phrase identification unit 73 of the third embodiment uses a learned model Mb to identify a practice phrase Z corresponding to the tendency data D. The learned model Mb is an example of a "second learned model."

[0075] As can be understood from the description of the first embodiment, there is a correlation between the performance tendency (tendency data D) of a performer and a practice phrase Z suitable for the performance tendency. For example, the practice phrase Z corresponding to each tendency data D is a piece of music suitable for improving the performance tendency specified by the tendency data D. The learned model Mb is a statistical estimation model that learns the relationship between the tendency data D and the practice phrase Z. The practice phrase identification unit 73 of the third embodiment inputs the tendency data D generated by the tendency identification unit 72 into the learned model Mb to identify a practice phrase Z corresponding to the performance tendency represented by the tendency data D. For example, the learned model Mb outputs an index of appropriateness for the tendency data D for each of a plurality of different practice phrases Z (i.e., the degree to which each practice phrase Z is appropriate for the performance tendency of the user U). The practice phrase identification unit 73 identifies the practice phrase Z with the largest index among the plurality of practice phrases Z stored in the storage device 22.

[0076] The trained model Mb is composed of, for example, a deep neural network. For example, any type of neural network, such as a recurrent neural network or a convolutional neural network, is used as the trained model Mb. The trained model Mb may be composed of a combination of multiple types of deep neural networks. In addition, additional elements such as a long short-term memory (LSTM) may be installed in the trained model Mb.

[0077] The learned model Mb is realized by a combination of a program that causes the control device 21 to execute a calculation to estimate the practice phrase Z from the tendency data D, and a plurality of variables (specifically, weights and biases) that are applied to the calculation. The program and the plurality of variables that realize the learned model Mb are stored in the storage device 22. The numerical values ​​of each of the plurality of variables that define the learned model Mb are set in advance by machine learning.

[0078] 15 is a flowchart illustrating a specific procedure of the specific process Sa in the third embodiment. The specific process Sa in the third embodiment is a process in which Sa3 in the specific process Sa in the first embodiment is replaced with step Sa23.

[0079] Acquisition of music piece data X and performance data Y by the performance data acquisition unit 71 (Sa1), and generation of trend data D by the trend identification unit 72 (Sa2) are the same as in the first embodiment. The practice phrase identification unit 73 of the third embodiment inputs the trend data D into the trained model Mb to identify practice phrase Z (Sa23). The process (Sa4) in which the practice phrase identification unit 73 transmits practice phrase Z to the electronic musical instrument 10 is the same as in the first embodiment.

[0080] The trained model Mb exemplified above is generated by the machine learning system 30. Fig. 16 is a block diagram illustrating a functional configuration related to generation of the trained model Mb in the machine learning system 30. The control device 31 executes a program stored in the storage device 32, thereby functioning as multiple elements (a training data acquisition unit 81b and a training processing unit 82b) for establishing the trained model Mb by machine learning.

[0081] The learning processing unit 82b establishes a learned model Mb by supervised machine learning (learning process Sd described later) using multiple pieces of learning data Tb. The learning data acquisition unit 81b acquires multiple pieces of learning data Tb. Specifically, the learning data acquisition unit 81b acquires multiple pieces of learning data Tb stored in the storage device 32 from the storage device 32. The learning data acquisition unit 81b is an example of a "second learning data acquisition unit," and the learning processing unit 82b is an example of a "second learning processing unit." Furthermore, the learning data Tb is an example of "second learning data."

[0082] Each of the multiple learning data Tb is composed of a combination of learning tendency data Dt and learning practice phrases Zt. The practice phrases Zt of each learning data Tb are pieces of music suitable for the performance tendency indicated by the tendency data Dt of the learning data Tb. The combination of the tendency data Dt and the practice phrases Zt is selected, for example, by the creator of the learning data T. The tendency data Dt is an example of "learning tendency data", and the practice phrases Zt are an example of "learning practice phrases".

[0083] 17 is a flowchart illustrating a specific procedure of the learning process Sd in which the control device 31 establishes the learned model Mb. The learning process Sd is also expressed as a method of generating the learned model Mb by machine learning (a method of generating a learned model).

[0084] When the learning process Sd is started, the learning data acquisition unit 81b selects one of the multiple learning data Tb (hereinafter referred to as "selected learning data Tb") stored in the storage device 32 (Sd1). As illustrated in Fig. 16, the learning processing unit 82b inputs the tendency data Dt of the selected learning data Tb to an initial or provisional model (hereinafter referred to as "provisional model Mb0") (Sd2), and acquires the training phrase Z estimated by the provisional model Mb0 in response to the input (Sd3).

[0085] The learning processing unit 82b calculates a loss function that represents an error between the training phrase Z estimated by the provisional model Mb0 and the training phrase Zt of the selected training data Tb (Sd4). The learning processing unit 82b updates multiple variables of the provisional model Mb0 so that the loss function is reduced (ideally minimized) (Sd5). For example, backpropagation is used to update the multiple variables according to the loss function.

[0086] The learning processing unit 82b determines whether a predetermined termination condition is satisfied (Sd6). If the termination condition is not satisfied (Sd6: NO), the learning processing unit 82b selects the unselected learning data Tb as new selected learning data Tb (Sd1). That is, the process of updating multiple variables of the provisional model Mb0 (Sd2-Sd5) is repeated until the termination condition is satisfied (Sd6: YES). The provisional model Mb0 at the time when the termination condition is satisfied (Sd6: YES) is determined as the trained model Mb.

[0087] As can be understood from the above explanation, the trained model Mb estimates a training phrase Z that is statistically valid for unknown trend data D based on the underlying relationship between trend data Dt and training phrases Zt in multiple training data Tb. In other words, the trained model Mb is a statistical estimation model that has learned the relationship between trend data D and training phrases Z. The training phrase identification unit 73 of the third embodiment identifies the training phrase Z by inputting trend data D to the trained model Mb that has learned the relationship between trend data Dt and training phrases Zt.

[0088] The learning processing unit 82b transmits the trained model Mb established by the above procedure to the information processing system 20 from the communication device 33 (Sd7). The control device 21 of the information processing system 20 stores the trained model Mb received from the machine learning system 30 in the storage device 22.

[0089] The third embodiment also achieves the same effects as the first embodiment. In the third embodiment, the practice phrase Z is identified by inputting the trend data D output by the trend identification unit 72 to the trained model Mb. Therefore, a statistically valid practice phrase Z can be identified based on the underlying relationship between the learning trend data Dt and the learning practice phrase Zt.

[0090] D: Fourth embodiment 18 is a block diagram illustrating a functional configuration of an electronic musical instrument 10 according to a fourth embodiment. In each of the above-described embodiments, a configuration in which the information processing system 20 includes a performance data acquisition unit 71, a tendency identification unit 72, and a practice phrase identification unit 73 has been exemplified. In the fourth embodiment, the electronic musical instrument 10 includes the performance data acquisition unit 71, the tendency identification unit 72, and the practice phrase identification unit 73. The above elements are realized by the control device 11 executing a program stored in the storage device 12. The control device 11 also functions as a presentation processing unit 74.

[0091] In addition to the multiple pieces of music piece data X similar to those in the first embodiment, a trained model Ma and multiple practice phrases Z are stored in the storage device 12 of the electronic musical instrument 10. The trained model Ma established by the machine learning system 30 is transferred to the electronic musical instrument 10, and the trained model Ma is stored in the storage device 12. Furthermore, each of the multiple practice phrases Z corresponds to a different set of tendency data D.

[0092] As in the first embodiment, the performance data acquisition unit 71 acquires performance data Y representing a performance of a piece of music by the user U, and music data X of the piece of music. Specifically, the performance data acquisition unit 71 generates the performance data Y in response to an operation from the user U to the performance device 14. The performance data acquisition unit 71 also acquires music data X of the piece of music to be performed by the user U from the storage device 12. The performance data acquisition unit 71 generates control data C including the music data X and the performance data Y.

[0093] As in the first embodiment, the tendency identification unit 72 generates tendency data D representing a performance tendency of the user U in response to the control data C. Specifically, the tendency identification unit 72 identifies the tendency data D by inputting the control data C including the music piece data X and the performance data Y to the trained model Ma.

[0094] As in the first embodiment, the practice phrase identification unit 73 uses the tendency data D identified by the tendency identification unit 72 to identify a practice phrase Z corresponding to the performance tendency of the user U. Specifically, the practice phrase identification unit 73 searches the storage device 12 for a practice phrase Z that corresponds to the tendency data D identified by the tendency identification unit 72 among the multiple practice phrases Z stored in the storage device 12.

[0095] The presentation processing unit 74 presents the practice phrase Z identified by the practice phrase identification unit 73 to the user U. Specifically, the presentation processing unit 74 causes the display device 15 to display the musical score of the practice phrase Z. The presentation processing unit 74 may also cause the playback system 18 to play back the performance sound of the practice phrase Z.

[0096] As can be understood from the above explanation, the fourth embodiment achieves the same effects as the first embodiment. Note that the configuration of the second embodiment in which the practice phrase identification unit 73 generates the practice phrase Z by editing the reference phrase Zref, and the configuration in which the practice phrase identification unit 73 identifies the practice phrase Z by using the trained model Mb are also applicable to the fourth embodiment in which the practice phrase identification unit 73 is mounted on the electronic musical instrument 10.

[0097] E: Fifth embodiment 19 is a block diagram illustrating the configuration of a performance system 100 according to the fifth embodiment. The performance system 100 includes an electronic musical instrument 10 and an information device 50. The information device 50 is, for example, a smartphone or a tablet terminal. The information device 50 is connected to the electronic musical instrument 10, for example, by wire or wirelessly.

[0098] The information device 50 is realized by a computer system including a control device 51 and a storage device 52. The control device 51 is composed of one or more processors that control each element of the information device 50. For example, the control device 51 is composed of one or more types of processors such as a CPU, an SPU, a DSP, an FPGA, or an ASIC. The storage device 52 is one or more memories that store programs executed by the control device 51 and various data used by the control device 51. The storage device 52 is composed of a known recording medium such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. Note that a portable recording medium that is detachable from the information device 50, or a recording medium (e.g., cloud storage) that the control device 51 can write or read via, for example, the communication network 200, may be used as the storage device 52.

[0099] The control device 51 executes the program stored in the storage device 52 to realize a performance data acquisition unit 71, a tendency identification unit 72, and a practice phrase identification unit 73. The configurations and operations of the performance data acquisition unit 71, the tendency identification unit 72, and the practice phrase identification unit 73 are similar to those illustrated in the first to fourth embodiments. Practice phrase Z identified by the practice phrase identification unit 73 is transmitted to the electronic musical instrument 10. The control device 11 of the electronic musical instrument 10 causes the display device 15 to display the musical score of practice phrase Z.

[0100] As will be understood from the above description, the fifth embodiment also achieves the same effects as the first to fourth embodiments. The information processing system 20 of the first to third embodiments, the electronic musical instrument 10 of the fourth embodiment, and the information device 50 of the fifth embodiment are examples of the "information processing system 20."

[0101] F: Variation Specific modified embodiments that can be added to each of the above-mentioned embodiments are illustrated below. A plurality of embodiments arbitrarily selected from the following examples may be appropriately combined as long as they are not mutually contradictory.

[0102] (1) In each of the above-described embodiments, the tendency data D is generated using one trained model Ma, but the tendency data D may be generated by selectively using multiple trained models Ma. For example, multiple trained models Ma corresponding to different musical instruments are prepared. The tendency identification unit 72 selects a trained model Ma corresponding to the musical instrument played by the user U from the multiple trained models Ma, and generates the tendency data D by inputting the control data C to the trained model Ma. The relationship between the content of the performance by the user U (performance data Y) and the performance tendency of the user U (trend data D) differs for each musical instrument. According to the configuration in which multiple trained models Ma corresponding to different musical instruments are selectively used, the tendency data D that appropriately represents the performance tendency of the musical instrument actually played by the user U can be generated.

[0103] (2) In the third embodiment, practice phrase Z is generated using one trained model Mb, but practice phrase Z may be generated by selectively using multiple trained models Mb. For example, multiple trained models Mb corresponding to different instruments are prepared. The practice phrase identification unit 73 selects a trained model Mb corresponding to the instrument played by the user U from the multiple trained models Mb, and generates practice phrase Z by inputting tendency data D to the trained model Mb.

[0104] (3) Any of a plurality of trained models Ma established by the machine learning system 30 may be selectively transferred to the electronic musical instrument 10 of the fourth embodiment. For example, of a plurality of trained models Ma corresponding to different musical instruments, a trained model Ma corresponding to an instrument designated by a user U of the electronic musical instrument 10 is transferred from the machine learning system 30 to the electronic musical instrument 10. Similarly, any of a plurality of trained models Ma established by the machine learning system 30 may be selectively transferred to the information device 50 of the fifth embodiment. In the third embodiment, any of a plurality of trained models Mb established by the machine learning system 30 may be selectively transferred to the information processing system 20.

[0105] (4) In each of the above-described embodiments, the indication data P is generated in response to an instruction from the instructor U2, but the control device 11 of the electronic musical instrument 10 may generate the indication data P in response to an instruction from the learner U1. For example, the learner U1 indicates a performance tendency (e.g., a playing style that is difficult) for his / her own performance and a time point at which the performance tendency is observed. The control device 11 generates the indication data P in response to an instruction from the user U, and transmits the indication data P from the communication device 13 to the machine learning system 30.

[0106] (5) In each of the above-mentioned embodiments, the control data C includes the music data X and the performance data Y. However, the contents of the control data C are not limited to the above-mentioned examples. For example, the control data C may include image data of an image captured of the user U playing the electronic musical instrument 10. For example, the control data C includes image data of both hands of the user U during performance. Similarly, the learning control data Ct includes image data of an image captured of the performer. According to the above-mentioned configuration, a suitable practice phrase Z that reflects the performance of the user U can be identified. In addition, a form in which the control data C does not include the music data X is also assumed. As can be understood from the above description, the control data C including at least the performance data Y is input to the trained model Ma. That is, the tendency identification unit 72 generates the tendency data D by inputting the performance data Y to the trained model Ma.

[0107] (6) In the first embodiment, a piece of music that is suitable for improving the performance tendencies of user U was given as an example of practice phrase Z. However, as in the second embodiment, the practice phrase identification unit 73 may identify a practice phrase Z that has a low level of difficulty in performance in the parts related to user U's performance tendencies.

[0108] (7) The configuration of the first embodiment in which one of a plurality of practice phrases Z is selected according to the trend data D may be combined with the configuration of the second embodiment in which the reference phrase Zref is edited according to the trend data D. For example, the practice phrase identification unit 73 selects one practice phrase Z according to the trend data D from among the plurality of practice phrases Z stored in the storage device 22 as the reference phrase Zref (Sa3), and generates the practice phrase Z by editing the reference phrase Zref according to the trend data D (Sa13). That is, the trend data D is shared between the selection of the practice phrase Z (Sa3) and the editing of the reference phrase Zref (Sa13).

[0109] (8) In the second embodiment, the practice phrase identification unit 73 generates practice phrase Z by editing one reference phrase Zref stored in the storage device 22, but practice phrase Z may be generated by selectively using multiple reference phrases Zref stored in the storage device 22. For example, the practice phrase generation unit may generate practice phrase Z by using a reference phrase Zref of a piece of music selected by a user U of the electronic musical instrument 10 from the multiple reference phrases Zref stored in the storage device 22.

[0110] (9) In each of the above-mentioned embodiments, an electronic keyboard instrument is exemplified as the electronic instrument 10, but the type of instrument played by the user U is arbitrary. For example, the user U may play an electric string instrument such as an electric guitar. An audio signal (audio data) representing the vibration of the strings of the electric string instrument, or MIDI format data generated by analyzing the musical tones produced by the electric string instrument, is used as the performance data Y. Examples of performance tendencies related to electric string instruments include tendencies such as "not sufficiently muting the part that should be muted" and "strings other than the string corresponding to the target note are sounding." For example, assuming that the user U plays a wind instrument such as a trumpet or a saxophone, the performance tendencies represented by the tendency data D are tendencies such as "unstable volume of musical tones" and "inaccurate pitch." For example, assuming that the user U plays a percussion instrument such as a drum, the performance tendencies represented by the tendency data D are tendencies such as "the time of striking is off" and "not good at hitting repeatedly at short intervals."

[0111] (10) In each of the above-described embodiments, a deep neural network is exemplified as the trained model Ma, but the trained model Ma is not limited to a deep neural network. For example, a statistical estimation model such as a hidden Markov model (HMM) or a support vector machine (SVM) may be used as the trained model Ma. The trained model Ma using the SVM will be described in detail below.

[0112] For example, an SVM is prepared for each of all combinations of two types of performance tendencies selected from multiple types of performance tendencies. For the SVM corresponding to the combination of two types of performance tendencies, a hyperplane in a multidimensional space is established by machine learning (learning process Sc). The hyperplane is a boundary surface that separates a space in which control data C corresponding to one of the two types of performance tendencies is distributed from a space in which control data C corresponding to the other performance tendency is distributed. The trained model Ma is composed of multiple SVMs corresponding to different combinations of performance tendencies (multi-class SVM).

[0113] The tendency identification unit 72 inputs the control data C to each of the multiple SVMs of the trained model Ma. The SVM corresponding to each combination selects one of two types of performance tendencies related to the combination depending on which of the two spaces separated by the hyperplane the control data C exists in. Selection of a performance tendency is similarly performed in each of the multiple SVMs corresponding to different combinations. The tendency identification unit 72 generates tendency data D representing the performance tendency that is selected the greatest number of times by the multiple SVMs among the multiple types of performance tendencies.

[0114] As can be understood from the above examples, regardless of the type of trained model Ma, the tendency identification unit 72 functions as an element that generates tendency data D representing the performance tendency of the user U by inputting the control data C into the trained model Ma. Note that while the above description focuses on the trained model Ma, a statistical estimation model such as HMM or SVM is also used for the trained model Mb of the third embodiment in a similar manner.

[0115] (11) In each of the above-described embodiments, supervised machine learning using a plurality of pieces of training data T is exemplified as the training process Sc, but the trained model Ma may be established by unsupervised machine learning that does not require training data T, or by reinforcement learning that maximizes rewards. An example of unsupervised machine learning is machine learning that uses known clustering. Similarly, the trained model Mb of the third embodiment may be established by unsupervised machine learning or reinforcement learning.

[0116] (12) In each of the above-described embodiments, the machine learning system 30 establishes the trained model Ma. However, the function (trained data acquisition unit 81a and learning processing unit 82a) by which the machine learning system 30 establishes the trained model Ma may be provided in the information processing system 20 of the first to third embodiments, the electronic musical instrument 10 of the fourth embodiment, or the information device 50 of the fifth embodiment. The same applies to the trained model Mb of the third embodiment. That is, the function (trained data acquisition unit 81b and learning processing unit 82b) by which the machine learning system 30 establishes the trained model Mb may be provided in the information processing system 20 of the third embodiment, the electronic musical instrument 10 of the fourth embodiment, or the information device 50 of the fifth embodiment.

[0117] (13) In each of the above-described embodiments, the learned model Ma is used to generate the tendency data D corresponding to the control data C, but the use of the learned model Ma may be omitted. For example, a table in which each of the multiple control data C and each of the multiple tendency data D are associated with each other may be used to generate the tendency data D. The table in which the correspondence between the control data C and the tendency data D is registered is stored in, for example, the storage device 22 of the first embodiment, the storage device 12 of the fourth embodiment, or the storage device 52 of the fifth embodiment. The tendency identification unit 72 searches the table for the tendency data D corresponding to the control data C generated by the performance data acquisition unit 71.

[0118] (14) In each of the above-described embodiments, a learned model Ma that has learned the relationship between the control data C including the music data X and the performance data Y and the tendency data D is used, but the configuration and method for generating the tendency data D from the control data C are not limited to the above examples. For example, a reference table in which the tendency data D is associated with each of a plurality of different control data C may be used for the tendency identification unit 72 to generate the tendency data D. The reference table is a data table in which the correspondence between the control data C and the tendency data D is registered, and is stored in, for example, the storage device 22 (the storage device 12 in the fourth embodiment). The tendency identification unit 72 searches the reference table for the control data C that corresponds to the combination of the music data X and the performance data Y, and acquires the tendency data D associated with the control data C from the reference table among the plurality of tendency data D.

[0119] (15) In the third embodiment, a trained model Mb that has learned the relationship between trend data D and practice phrases Z is used, but the configuration and method for generating practice phrases Z from trend data D are not limited to the above examples. For example, a reference table in which practice phrases Z are associated with each of a plurality of different trend data D may be used for generating practice phrases Z by the practice phrase identification unit 73. The reference table is a data table in which the correspondence between trend data D and practice phrases Z is registered, and is stored in, for example, the storage device 22 (storage device 12 in the fourth embodiment). The practice phrase identification unit 73 searches the reference table for the practice phrase Z corresponding to the trend data D, and obtains the practice phrase Z associated with the trend data D from the reference table among the plurality of practice phrases Z.

[0120] (16) In each of the above-described embodiments, the performance data acquisition unit 71 acquires the performance data Y representing the performance of the user U from the electronic musical instrument 10, but the method by which the performance data acquisition unit 71 acquires the performance data Y is not limited to the above examples. For example, it is not necessary for the performance data acquisition unit 71 to acquire the performance data Y in real time in parallel with a performance on the performance device 14. For example, the performance data acquisition unit 71 may acquire performance data Y that records a past performance by the user U from the electronic musical instrument 10. In other words, in the present disclosure, it is not important whether the performance data acquisition unit 71 acquires the performance data Y in real time in relation to a performance by the user U.

[0121] Also, for example, the performance data acquisition unit 71 does not need to receive the performance data Y representing the string of notes played by the user U from the electronic musical instrument 10. For example, the performance data acquisition unit 71 may receive video data capturing the performance of the user U via the communication device 23 and generate the performance data Y by analyzing the video data. In other words, the "acquisition" of the performance data Y by the performance data acquisition unit 71 includes not only the process of receiving the performance data Y from an external device such as the electronic musical instrument 10, but also the process of generating the performance data Y from information such as the video data.

[0122] (17) In each of the above-mentioned embodiments, the learning data acquisition unit 81a acquires the performance data Y0 representing the performance of a piece of music by the learner U1 and the instruction data P representing the instruction by the instructor U2, but the method by which the learning data acquisition unit 81a acquires the learning data Ta is not limited to the above examples. For example, the learning data acquisition unit 81a does not need to acquire the performance data Y0 and the instruction data P (and further the learning data Ta) in parallel with the performance by the learner U1 and the instruction by the instructor U2. For example, the learning data acquisition unit 81a may acquire the performance data Y0 recording the past performance by the learner U1 and the instruction data P recording the past instruction by the instructor U2. In other words, in the present disclosure, it is not important whether the learning data acquisition unit 81a acquires the performance data Y0 and the instruction data P in real time in response to the performance by the learner U1 and the instruction by the instructor U2.

[0123] Also, for example, the learning data acquisition unit 81a does not need to receive the performance data Y0 representing the string of notes played by the learner U1 from the electronic musical instrument 10. For example, the learning data acquisition unit 81a may receive video data of the performance of the learner U1 via the communication device 23 and generate the performance data Y0 by analyzing the video data. In other words, the "acquisition" of the performance data Y0 by the learning data acquisition unit 81a includes not only the process of receiving the performance data Y0 from an external device such as the electronic musical instrument 10, but also the process of generating the performance data Y0 from information such as the video data.

[0124] Similarly, the learning data acquisition unit 81a does not need to receive the instruction data P representing the instruction by the instructor U2 from the information device 40. For example, the learning data acquisition unit 81a may receive video data of the instruction by the instructor U2 via the communication device 23 and generate the instruction data P by analyzing the video data. In other words, the "acquisition" of the instruction data P by the learning data acquisition unit 81a includes a process of receiving the instruction data P from an external device such as the information device 40, as well as a process of generating the instruction data P from information such as the video data.

[0125] (18) In each of the above-described embodiments, the learning data acquisition unit 81a extracts, as the performance data Yt, a portion of the performance data Y0 transmitted from the electronic musical instrument 10 within a specific section including a time point designated by the time data τ of the instruction data P. However, the learning performance data Yt may be transmitted from the electronic musical instrument 10 to the machine learning system 30. For example, the control device 11 of the electronic musical instrument 10 receives the instruction data P from the information device 40, and transmits, as the performance data Yt, a portion of the performance data Y0 within a specific section corresponding to the time data τ of the instruction data P from the communication device 13 to the machine learning system 30. The learning data acquisition unit 81a receives the performance data Yt transmitted from the electronic musical instrument 10 by the communication device 33. According to the above configuration, the machine learning system 30 does not need to acquire the time data τ from the information device 40. In other words, the time data τ may be omitted from the instruction data P transmitted from the information device 40 to the machine learning system 30.

[0126] Although the above description focuses on the performance data Yt, the learning music piece data Xt may also be transmitted from the electronic musical instrument 10 to the machine learning system 30. For example, the control device 11 of the electronic musical instrument 10 transmits a portion of the music piece data X0 within a specific section that corresponds to the time data τ of the indication data P as music piece data Xt from the communication device 13 to the machine learning system 30. The learning data acquisition unit 81a receives the music piece data Xt transmitted from the electronic musical instrument 10 via the communication device 33.

[0127] (19) The functions exemplified in each of the above-mentioned embodiments (the performance data acquisition unit 71, the tendency identification unit 72, and the practice phrase identification unit 73) are realized by the cooperation of one or more processors constituting the control device and the programs stored in the storage device, as described above. The above-mentioned programs can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that the non-transitory recording medium includes any recording medium except for a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via the communication network 200, the recording medium that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

[0128] G: Notes From the above-described exemplary embodiments, the following configurations can be understood, for example.

[0129] The information processing system according to one aspect (aspect 1) includes a performance data acquisition unit that acquires performance data representing a performance of a musical piece by a user, a tendency identification unit that generates tendency data representing a performance tendency of the user by inputting the performance data acquired by the performance data acquisition unit into a first trained model that has learned the relationship between learning performance data representing a performance of a reference musical piece and learning tendency data representing a performance tendency represented by the learning performance data, and a practice phrase identification unit that identifies a practice phrase according to the tendency data generated by the tendency identification unit. According to the above aspect, by inputting the performance data representing a performance of a musical piece by a user into the first trained model, tendency data representing a performance tendency of the user is generated, and a practice phrase according to the performance tendency of the user is identified according to the tendency data. Therefore, effective practice according to the performance tendency of the user is realized by playing the practice phrase.

[0130] "Performance data" is data in any format that represents a performance by a user. For example, examples of performance data include music data (e.g., MIDI data) that represents a time series of notes played by the user, and audio data that represents the sounds produced by an instrument played by the user. Furthermore, video data that captures the user's performance may be included in the performance data.

[0131] The "tendency data" is data in any format that indicates the performance tendency of the user. The "performance tendency" is, for example, the tendency of the user to make performance mistakes or the tendency of a playing style that the user is not good at. For example, the tendency data specifies one of a plurality of tendencies regarding performance mistakes or playing styles.

[0132] A "practice phrase" is a sequence of notes (melody) for a user to practice playing. A "practice phrase according to the user's playing tendencies" is, for example, a sequence of notes suitable for overcoming playing mistakes that tend to occur in the user's playing or a playing style that the user is not good at. A practice phrase may be an entire piece of music or a part of the piece of music.

[0133] In a specific example (Aspect 2) of Aspect 1, the first trained model is a model that has learned a relationship between the learning control data, which includes the learning music piece data representing the score of the reference music piece and the learning performance data, and the learning tendency data, and the tendency identification unit generates the tendency data by inputting control data, which includes the performance data and music piece data representing the score of the music piece, to the first trained model. According to the above aspect, since the control data includes the music piece data in addition to the performance data, appropriate tendency data that reflects the relationship (e.g., similarity or difference) between the performance data and the music piece data can be generated.

[0134] In a specific example (Aspect 3) of Aspect 1 or Aspect 2, the practice phrase identification unit selects a practice phrase corresponding to the tendency represented by the tendency data from among a plurality of practice phrases corresponding to different performance tendencies. According to the above aspect, since a practice phrase corresponding to the user's performance tendency is selected from a plurality of practice phrases, the processing load of the practice phrase identification unit in identifying a practice phrase is reduced.

[0135] In a specific example (Aspect 4) of Aspect 1 or Aspect 2, the practice phrase identification unit generates the practice phrase by editing a reference phrase according to the tendency represented by the tendency data. According to the above aspect, since a practice phrase is generated by editing a reference phrase, it is possible to provide the user with a practice phrase appropriate to the user's level of playing technique.

[0136] "Editing the reference phrase" refers to a process of changing the reference phrase so that the difficulty of playing the phrase changes according to the tendency represented by the tendency data. Examples of "editing" include simplifying the chords in the reference phrase (e.g., omitting the constituent notes of the chord), omitting a leap progression (a part in which two notes with a large pitch difference are played in succession), or simplifying the fingering when playing the phrase.

[0137] In a specific example of aspect 4 (aspect 5), the reference phrase includes a time sequence of chords, and editing the reference phrase includes changing the chords. In another specific example of aspect 4 (aspect 6), the reference phrase includes a leap progression in which the pitch difference exceeds a predetermined value, and editing the reference phrase includes omitting or changing the leap progression. In another specific example of aspect 4 (aspect 7), the reference phrase includes a specification of a playing style of an instrument, and editing the reference phrase includes changing the playing style. "Playing style" refers to the way an instrument is played. For example, fingering in a keyboard instrument or string instrument, or special playing styles such as hammering, pulling, or cutting in a string instrument such as a guitar or bass are exemplified as "playing styles".

[0138] In a specific example (Aspect 8) of any one of Aspects 1 to 4, the practice phrase identification unit identifies the practice phrase by inputting the tendency data output by the tendency identification unit into a second trained model that has learned the relationship between learning tendency data representing a performance tendency and learning practice phrases corresponding to the tendency represented by the learning tendency data. According to the above aspect, the practice phrase identification unit identifies a practice phrase by inputting the tendency data output by the tendency identification unit into the second trained model. Therefore, a practice phrase that is statistically valid can be identified based on the underlying relationship between the learning tendency data and the learning practice phrase.

[0139] In a specific example (aspect 9) of aspect 8, the practice phrase identification unit identifies the practice phrase by selectively using one of a plurality of second trained models corresponding to different instruments. According to the above aspect, compared to a configuration that uses only one second trained model, it is possible to identify a practice phrase appropriate for the instrument that the user actually plays.

[0140] In a specific example (Aspect 10) of any of Aspects 1 to 9, the tendency identification unit selectively uses one of a plurality of first trained models corresponding to different musical instruments to generate the tendency data. According to the above aspect, since a plurality of first trained models corresponding to different musical instruments are selectively used to generate the tendency data, it is possible to generate tendency data that appropriately represents the playing tendency of the musical instrument actually played by the user, compared to a configuration that uses only one first trained model.

[0141] An electronic musical instrument according to one aspect (aspect 11) of the present disclosure includes a performance receiving unit that accepts a musical piece performance by a user, a performance data acquisition unit that acquires performance data representing the performance accepted by the performance receiving unit, a tendency identification unit that outputs tendency data representing the performance tendency of the user from the first learned model by inputting the performance data acquired by the performance data acquisition unit into a first learned model that has learned the relationship between learning performance data representing the musical piece performance and learning tendency data that represents the performance tendency represented by the learning performance data, a practice phrase identification unit that uses the tendency data output by the tendency identification unit to identify a practice phrase corresponding to the playing tendency of the user, and a presentation processing unit that presents the practice phrase to the user.

[0142] The presentation processing unit presents the practice phrase to the user in a manner that the user can perceive visually or audibly. For example, an element that displays the musical score of the practice phrase on a display device, or an element that causes a sound emitting device to emit the performance sound of the practice phrase, are exemplified as the presentation processing unit.

[0143] An information processing method according to one aspect (aspect 12) of the present disclosure acquires performance data representing a musical piece played by a user, and inputs the acquired performance data into a first trained model that has learned the relationship between learning performance data representing the musical piece performance and learning tendency data that represents the performance tendency represented by the learning performance data, thereby generating tendency data that represents the playing tendency of the user, and identifying practice phrases corresponding to the tendency data.

[0144] In a specific example (Aspect 13) of Aspect 12, the practice phrase is identified by selecting a practice phrase corresponding to the tendency represented by the tendency data from among a plurality of practice phrases corresponding to different tendencies of performance. Also, in a specific example (Aspect 14) of Aspect 12, the practice phrase is generated by editing a reference phrase according to the tendency represented by the tendency data. In another specific example (Aspect 15) of Aspect 12, the practice phrase is identified by inputting the tendency data into a second trained model that has learned the relationship between learning tendency data representing performance tendencies and learning practice phrases corresponding to the tendencies represented by the learning tendency data.

[0145] A machine learning system according to one aspect (aspect 16) of the present disclosure includes a first learning data acquisition unit that acquires first learning data including learning performance data representing a performance of a musical piece by a user and learning tendency data representing a tendency of the performance represented by the instruction data, and a first learning processing unit that establishes a first trained model that learns a relationship between the learning performance data and the learning tendency data by machine learning using the first learning data. According to the above aspect, it is possible to generate tendency data that is statistically valid for the performance data by the first trained model under the latent relationship between the learning performance data and the learning tendency data.

[0146] In a specific example (aspect 17) of aspect 16, the first learning data acquisition unit acquires performance data representing a performance of the musical piece by the user and comment data representing a time point in the musical piece and a tendency of the performance at that time point, and generates the first learning data including the learning performance data representing a performance within a section of the performance data including the time point represented by the comment data, and the learning tendency data representing the tendency of the performance represented by the comment data. According to the above aspect, it is not necessary for a source of the performance data (e.g., the first device) to extract a section of the performance by the user that corresponds to the time point represented by the comment data.

[0147] In a specific example (Aspect 18) of Aspect 17, the first learning data acquisition unit acquires the performance data from a first device, and acquires the instruction data from a second device separate from the first device. According to the above aspect, data for machine learning can be prepared by using data (performance data and instruction data) acquired from, for example, a first device and a second device located remotely from each other. The first device is, for example, a terminal device used by a practitioner who practices playing a musical instrument, and the second device is, for example, a terminal device used by an instructor who evaluates and instructs the performance of the practitioner.

[0148] In a specific example (Aspect 19) of any of Aspects 16 to 18, the first trained model is a model that has trained the relationship between the learning control data, which includes the learning music piece data representing the musical score of the reference music piece and the learning performance data, and the learning tendency data. In the above aspects, since the learning control data includes the learning music piece data in addition to the learning performance data, it is possible to establish a first trained model that can generate appropriate tendency data that reflects the relationship (e.g., similarity or difference) between the learning performance data and the learning music piece data.

[0149] In a specific example (aspect 20) of any of aspects 16 to 19, the system further includes a second learning data acquisition unit that acquires a plurality of second learning data including learning tendency data representing a performance tendency and learning practice phrases corresponding to the tendency represented by the learning tendency data, and a second learning processing unit that establishes a second trained model that learns the relationship between the learning tendency data and the learning practice phrases in the second learning data by machine learning using the plurality of second learning data.

[0150] A machine learning method according to one aspect (aspect 21) of the present disclosure acquires performance data representing a performance of a musical piece by a user and comment data representing a time point within the musical piece and a tendency of the performance at that time point, and establishes a first trained model that learns the relationship between the training performance data and the training tendency data by machine learning using first training data including training performance data representing a performance within a section of the performance data that includes the time point represented by the comment data, and training tendency data representing the tendency of the performance represented by the comment data. [Explanation of symbols]

[0151] 100...performance system, 10...electronic musical instrument, 11,21,31,41,51...control device, 12,22,32,42,52...storage device, 13,23,33,43...communication device, 14...performance device, 15,45...display device, 16...sound source device, 17...sound emission device, 18,46...playback system, 20...information processing system, 30...machine learning system, 40...information device, 44...operation device, 50...information device, 71...performance data acquisition unit, 72...trend identification unit, 73...practice phrase identification unit, 74...presentation processing unit, 81a,81b...learning data acquisition unit, 82a,82b...learning processing unit.

Claims

1. a receiving unit for receiving performance data representing a performance of a piece of music by a learner; a generating unit that generates tendency data representing a performance tendency designated by an instructor for the performance using the performance data; An information processing system comprising:

2. A display unit that displays the performance tendency represented by the tendency data. The information processing system of claim 1 further comprising:

3. The generating unit generates the tendency data representing a performance tendency selected by the instructor from a plurality of options.

2. The information processing system of claim 1.

4. a learning data acquisition unit that acquires a plurality of learning data each including the performance data and the tendency data; a learning processing unit that establishes a learned model that learns a relationship between the performance data and the tendency data in the plurality of learning data by machine learning using the plurality of learning data; The information processing system of claim 1 further comprising:

5. each of the plurality of learning data includes control data including music piece data representing a score of the music piece and the performance data, and the tendency data; The trained model is a model that has trained the relationship between the control data and the trend data.

5. The information processing system of claim 4.

6. a transmission unit for transmitting performance data representing a performance of a piece of music by a learner; a receiving unit that receives tendency data representing a performance tendency designated by an instructor for the performance using the performance data; An electronic musical instrument comprising:

7. receiving performance data representing a performance of a piece of music by a practitioner; Using the performance data, tendency data is generated that represents a performance tendency designated by an instructor regarding the performance. An information processing method implemented by a computer system.

8. a learning data acquisition unit that acquires a plurality of learning data each including performance data representing a performance of a piece of music by a learner and tendency data representing a performance tendency designated by an instructor for said performance using said performance data; a learning processing unit that establishes a learned model that learns a relationship between the performance data and the tendency data in the plurality of learning data by machine learning using the plurality of learning data; A machine learning system comprising:

Images